2024-04-25T01:09:28.811 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:09:28.811 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312 -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/orig.config.yaml 2024-04-25T01:09:28.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:09:29.043 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672312' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBD60tGfySjDhn0VHJ9lG6uCTsEtqhTutatrjZF5pph2nv/Qb1R3/NXo0Mr+Qi0kkqmxjCttZCA3XQAecjuNfQ= smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvIiYtODVZAPMWcmIP6ZSkiM9hB0yD8e8EtB29x0TAZYHy/+Ez1z4pgEJErcgR/VU50sI8poC+LGYnAwdUcpF0= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZoHTub48qYtoNM0CtL9R7EpTXxBNGoYZQsv4sd2jECdyHSeYbBHi1LS15RVz6FDItnSkiWIE2GZvRFEcn+r2U= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:09:29.043 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:09:29.044 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:09:29.044 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:09:29.046 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:09:29.070 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:09:29.070 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:09:29.070 INFO:teuthology.packaging:ref: None 2024-04-25T01:09:29.070 INFO:teuthology.packaging:tag: None 2024-04-25T01:09:29.070 INFO:teuthology.packaging:branch: squid 2024-04-25T01:09:29.070 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:09:29.071 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:09:29.271 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:09:29.273 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:09:29.280 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:09:29.280 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:09:29.299 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:09:29.318 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:09:29.324 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:09:29.341 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:04:32.024342', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBD60tGfySjDhn0VHJ9lG6uCTsEtqhTutatrjZF5pph2nv/Qb1R3/NXo0Mr+Qi0kkqmxjCttZCA3XQAecjuNfQ='} 2024-04-25T01:09:29.359 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:04:32.026076', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGvIiYtODVZAPMWcmIP6ZSkiM9hB0yD8e8EtB29x0TAZYHy/+Ez1z4pgEJErcgR/VU50sI8poC+LGYnAwdUcpF0='} 2024-04-25T01:09:29.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:04:32.022467', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZoHTub48qYtoNM0CtL9R7EpTXxBNGoYZQsv4sd2jECdyHSeYbBHi1LS15RVz6FDItnSkiWIE2GZvRFEcn+r2U='} 2024-04-25T01:09:29.380 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:09:29.386 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T01:09:29.386 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T01:09:29.386 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T01:09:29.387 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:09:29.485 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f629f709160>, signals=[15]) 2024-04-25T01:09:29.486 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:09:29.492 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:09:29.492 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:09:29.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:09:29.582 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-25T01:09:29.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:09:29.657 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-04-25T01:09:29.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:09:29.733 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:09:29.740 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-25T01:09:29.759 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-25T01:09:29.760 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-25T01:09:29.818 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-04-25T01:09:29.818 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-04-25T01:09:29.818 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-04-25T01:09:29.818 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:09:29.818 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-04-25T01:09:29.818 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:09:29.819 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:09:29.822 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-25T01:09:29.849 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-25T01:09:29.867 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-25T01:09:29.867 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-25T01:09:29.925 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:09:29.926 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:09:29.926 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:09:29.926 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:09:29.926 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:09:29.926 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:09:29.926 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-25T01:09:29.957 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-04-25T01:09:29.975 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-04-25T01:09:29.976 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:09:30.033 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:09:30.034 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:09:30.035 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-04-25T01:09:30.057 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:09:30.068 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:09:30.073 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:09:30.073 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:09:30.076 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:09:30.078 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:09:30.095 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:09:30.100 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:09:30.100 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:09:30.133 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:09:30.136 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:09:30.162 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:09:30.241 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:09:30.259 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:09:30.259 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-25T01:09:30.259 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:09:30.260 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:09:30.260 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:09:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:09:30.277 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-04-25T01:09:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:09:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:09:30.279 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-25T01:09:30.280 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-04-25T01:09:30.334 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:09:30.335 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-437.el9.x86_64 2024-04-25T01:09:30.335 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-04-25T01:09:30.337 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:09:30.337 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-437.el9.x86_64 2024-04-25T01:09:30.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-04-25T01:09:30.338 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:09:30.338 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-437.el9.x86_64 2024-04-25T01:09:30.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-04-25T01:09:32.567 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T01:09:33.381 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-04-25T01:09:36.666 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 19 MB 00:02 2024-04-25T01:09:41.082 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 910 kB/s | 8.0 MB 00:09 2024-04-25T01:09:41.175 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.2 MB 00:00 2024-04-25T01:09:42.875 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-25T01:09:43.383 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-25T01:09:44.027 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-25T01:09:49.141 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-04-25T01:09:49.319 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-25T01:09:50.175 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-04-25T01:09:51.250 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:09:51.250 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:09:51.332 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-25T01:09:51.472 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-25T01:09:51.472 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-25T01:09:51.473 DEBUG:teuthology.orchestra.run.smithi038:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:09:51.702 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-25T01:09:51.995 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:09:49 AM UTC. 2024-04-25T01:09:52.129 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-25T01:09:52.138 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-25T01:09:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:Operation aborted. 2024-04-25T01:09:52.195 INFO:teuthology.orchestra.run.smithi038.stdout:Is this ok [y/N]: 2024-04-25T01:09:52.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum reinstall -y kernel || true 2024-04-25T01:09:52.714 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:09:49 AM UTC. 2024-04-25T01:09:52.835 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-25T01:09:52.836 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-25T01:09:52.836 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-25T01:09:52.836 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-25T01:09:52.836 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-25T01:09:52.837 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-25T01:09:53.263 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64.rpm 854 kB/s | 255 kB 00:00 2024-04-25T01:09:53.263 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:09:53.263 INFO:teuthology.orchestra.run.smithi038.stdout:Total 598 kB/s | 255 kB 00:00 2024-04-25T01:09:53.282 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-25T01:09:53.481 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-25T01:09:53.481 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-25T01:09:53.486 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-25T01:09:53.487 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-25T01:09:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-25T01:09:53.626 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-25T01:09:53.714 INFO:teuthology.orchestra.run.smithi038.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:09:53.714 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:09:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:09:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:09:55.075 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:09:55.076 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:09:55.076 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalled: 2024-04-25T01:09:55.076 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:09:55.076 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:09:55.076 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-25T01:09:55.236 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:09:55.284 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:09:55.284 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:09:55.285 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:09:55.285 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T01:09:55.285 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T01:09:55.321 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-04-25T01:09:55.346 DEBUG:teuthology.parallel:result is None 2024-04-25T01:09:56.733 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.2 MB 00:02 2024-04-25T01:09:58.184 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-04-25T01:09:58.601 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T01:09:59.879 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-25T01:10:00.466 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:10:00.466 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:10:00.542 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-25T01:10:00.543 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-04-25T01:10:00.543 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-25T01:10:00.658 DEBUG:teuthology.orchestra.run.smithi156:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:10:01.157 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:09:58 AM UTC. 2024-04-25T01:10:01.286 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-25T01:10:01.295 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-25T01:10:01.295 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-04-25T01:10:01.295 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-25T01:10:01.295 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalling: 2024-04-25T01:10:01.295 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 255 k 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2024-04-25T01:10:01.296 INFO:teuthology.orchestra.run.smithi156.stderr:Operation aborted. 2024-04-25T01:10:01.364 INFO:teuthology.orchestra.run.smithi156.stdout:Is this ok [y/N]: 2024-04-25T01:10:01.364 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum reinstall -y kernel || true 2024-04-25T01:10:01.861 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:09:58 AM UTC. 2024-04-25T01:10:01.977 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-25T01:10:01.978 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-25T01:10:01.978 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-04-25T01:10:01.978 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-25T01:10:01.978 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalling: 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 255 k 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2024-04-25T01:10:01.979 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-04-25T01:10:02.253 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-04-25T01:10:02.253 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:10:02.253 INFO:teuthology.orchestra.run.smithi156.stdout:Total 929 kB/s | 255 kB 00:00 2024-04-25T01:10:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-04-25T01:10:02.462 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-04-25T01:10:02.462 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-04-25T01:10:02.467 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-04-25T01:10:02.467 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-04-25T01:10:02.609 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-04-25T01:10:02.687 INFO:teuthology.orchestra.run.smithi156.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:10:02.688 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:10:03.331 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:10:03.331 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:10:04.053 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:10:04.053 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:10:04.053 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalled: 2024-04-25T01:10:04.053 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:10:04.053 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:10:04.053 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-25T01:10:04.250 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:10:04.296 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:10:04.297 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:10:04.297 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:10:04.297 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-25T01:10:04.297 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T01:10:04.324 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-04-25T01:10:04.342 DEBUG:teuthology.parallel:result is None 2024-04-25T01:10:05.068 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T01:10:07.016 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:10:07.016 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:10:07.121 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-25T01:10:07.123 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-25T01:10:07.123 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-25T01:10:07.215 DEBUG:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:10:07.777 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 01:10:05 AM UTC. 2024-04-25T01:10:07.909 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-25T01:10:07.916 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-25T01:10:07.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:10:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 255 k 2024-04-25T01:10:07.917 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-04-25T01:10:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:Operation aborted. 2024-04-25T01:10:07.971 INFO:teuthology.orchestra.run.smithi016.stdout:Is this ok [y/N]: 2024-04-25T01:10:07.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2024-04-25T01:10:08.487 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:10:05 AM UTC. 2024-04-25T01:10:08.607 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-25T01:10:08.608 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-25T01:10:08.608 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 255 k 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-04-25T01:10:08.609 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-25T01:10:09.145 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-437.el9.x86_64.rpm 984 kB/s | 255 kB 00:00 2024-04-25T01:10:09.146 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:10:09.146 INFO:teuthology.orchestra.run.smithi016.stdout:Total 475 kB/s | 255 kB 00:00 2024-04-25T01:10:09.163 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-25T01:10:09.360 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-25T01:10:09.360 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-25T01:10:09.365 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-25T01:10:09.366 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-25T01:10:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-25T01:10:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:10:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:10:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:10:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:10:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:10:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:10:10.941 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalled: 2024-04-25T01:10:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:10:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:10:10.941 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-25T01:10:11.119 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:10:11.168 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:10:11.168 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:10:11.168 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:10:11.168 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T01:10:11.168 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T01:10:11.201 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-04-25T01:10:11.225 DEBUG:teuthology.parallel:result is None 2024-04-25T01:10:11.225 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:10:11.232 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:10:11.232 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:10:11.267 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:10:11.270 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:10:11.288 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:10:11.295 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:10:11.302 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:10:11.302 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:10:11.322 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:10:11.329 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:10:11.365 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:10:11.372 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:10:11.372 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:10:11.387 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:10:11.396 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:10:11.411 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:10:11.421 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:10:11.428 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:10:11.438 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:10:11.441 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:10:11.448 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:10:11.450 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:10:11.458 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:10:11.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:10:11.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:10:11.486 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:10:11.517 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:10:11.526 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:10:11.526 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:10:11.530 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:10:11.554 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:10:11.573 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:10:11.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:10:11.669 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:10:11.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:10:11.790 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:10:11.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:10:11.855 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:10:11.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:10:11.952 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:10:12.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:10:12.083 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:10:12.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:10:12.152 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:10:12.195 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:10:12.257 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:10:12.298 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:10:12.341 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:10:12.341 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:10:12.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-25T01:10:12.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-25T01:10:12.413 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-04-25T01:10:12.446 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:10:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:10:12.487 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:10:12.907 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:10:12.915 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:10:12.915 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:10:12.924 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:10:12.931 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:10:12.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-25T01:10:12.997 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-25T01:10:12.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-25T01:10:13.069 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-25T01:10:13.071 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-04-25T01:10:13.106 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-04-25T01:10:13.108 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:10:13.108 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-25T01:10:13.129 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-25T01:10:13.130 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-25T01:10:13.151 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-25T01:10:13.152 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-04-25T01:10:13.180 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-04-25T01:10:13.180 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:10:13.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:10:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:10:13.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:10:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:10:13.232 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:10:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:10:13.255 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:10:13.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:10:13.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:10:13.329 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:10:13.353 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:10:13.363 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T01:10:13.368 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T01:10:13.488 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:10:13.504 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:10:13.506 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi038.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:13:49.549 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2024-04-25T01:13:49.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-25T01:13:49.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:13:49.626 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-25T01:13:49.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-25T01:13:49.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-25T01:13:49.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:13:49.787 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-25T01:13:49.871 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-25T01:13:49.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-25T01:13:49.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:13:49.946 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-04-25T01:13:50.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-25T01:13:50.022 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:13:50.036 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:13:50.036 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:13:50.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:13:50.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:13:50.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:13:50.042 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:13:50.042 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:13:50.071 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:13:50.073 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:13:50.087 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:13:50.088 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:13:50.089 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:13:50.102 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:13:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-25T01:13:50.115 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-25T01:13:50.124 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-04-25T01:13:50.124 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-25T01:13:50.126 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-25T01:13:50.134 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-04-25T01:13:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:13:50.141 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:13:50.147 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:13:50.152 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:13:50.155 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:13:50.160 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:13:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:13:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:13:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-25T01:13:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.212 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:13:50.213 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:13:50.214 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:13:50.214 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-25T01:13:50.214 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.214 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.214 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.214 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.216 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:13:50.216 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-25T01:13:50.216 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.216 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.216 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.216 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:50.217 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:13:50.226 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:13:50.226 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:13:50.226 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:13:50.226 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:13:50.237 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:13:50.237 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:13:50.238 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:13:50.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:50.240 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:13:50.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:50.241 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:13:50.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:50.474 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:13:50.474 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:13:50.479 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:13:50.479 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:13:50.482 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:13:50.482 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:13:50.589 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:13:50.589 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:13:50.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:13:50.615 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:13:50.616 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:13:50.643 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:13:50.643 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:13:50.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:13:50.650 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:13:50.650 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:13:50.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:13:50.672 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:13:50.672 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:13:50.677 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:13:50.678 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:13:50.680 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:13:50.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:13:50.744 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:13:50.753 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:13:50.815 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:13:50.816 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-04-25T01:13:50.817 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-04-25T01:13:50.820 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:13:50.880 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-04-25T01:13:50.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-04-25T01:13:50.885 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-25T01:13:50.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-25T01:13:51.043 INFO:teuthology.orchestra.run.smithi156.stdout:48 files removed 2024-04-25T01:13:51.070 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:13:51.117 INFO:teuthology.orchestra.run.smithi016.stdout:48 files removed 2024-04-25T01:13:51.120 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-25T01:13:51.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:13:51.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:13:51.765 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 239 kB/s | 83 kB 00:00 2024-04-25T01:13:51.791 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 182 kB/s | 83 kB 00:00 2024-04-25T01:13:51.883 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-04-25T01:13:52.124 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-04-25T01:13:52.191 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-25T01:13:52.301 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-04-25T01:13:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-25T01:13:52.567 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-04-25T01:13:52.735 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-25T01:13:52.983 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-25T01:13:53.443 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-25T01:13:54.659 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-04-25T01:14:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-04-25T01:14:03.227 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-04-25T01:14:04.227 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-25T01:14:05.761 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-25T01:14:07.328 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-04-25T01:14:07.412 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T01:14:09.029 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-25T01:14:10.417 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-25T01:14:12.541 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-04-25T01:14:12.820 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T01:14:15.014 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:14:15.014 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:14:15.026 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:14:15.026 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:14:15.027 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:14:15.028 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:14:15.030 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:14:15.032 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:14:15.033 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:14:15.098 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:14:15.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:14:15.109 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:14:15.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:14:15.111 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:14:15.112 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:14:15.113 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:14:15.114 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:14:15.115 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:14:15.116 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:14:15.117 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:14:15.118 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:14:15.119 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout:Install 148 Packages 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 4 Packages 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:14:15.120 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 220 M 2024-04-25T01:14:15.121 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-25T01:14:15.623 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-04-25T01:14:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-04-25T01:14:16.164 INFO:teuthology.orchestra.run.smithi016.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 3.6 MB/s | 890 kB 00:00 2024-04-25T01:14:16.247 INFO:teuthology.orchestra.run.smithi016.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.7 MB/s | 145 kB 00:00 2024-04-25T01:14:16.489 INFO:teuthology.orchestra.run.smithi016.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 7.2 MB/s | 5.4 MB 00:00 2024-04-25T01:14:16.563 INFO:teuthology.orchestra.run.smithi016.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 7.5 MB/s | 2.4 MB 00:00 2024-04-25T01:14:16.840 INFO:teuthology.orchestra.run.smithi016.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 4.3 MB/s | 1.5 MB 00:00 2024-04-25T01:14:16.975 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.2 MB 00:00 2024-04-25T01:14:17.040 INFO:teuthology.orchestra.run.smithi016.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-04-25T01:14:17.833 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:14:17.834 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:14:17.845 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:14:17.846 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:14:17.847 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:14:17.847 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:14:17.850 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:14:17.851 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:14:17.853 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:14:17.919 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-25T01:14:17.923 INFO:teuthology.orchestra.run.smithi016.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 9.6 MB/s | 21 MB 00:02 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:14:17.929 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:14:17.930 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:14:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-25T01:14:17.932 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:14:17.933 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:14:17.934 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:14:17.935 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:14:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:14:17.938 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:14:17.939 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:14:17.940 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:14:17.941 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-25T01:14:17.942 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Install 148 Packages 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 4 Packages 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:14:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 220 M 2024-04-25T01:14:17.944 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-25T01:14:18.049 INFO:teuthology.orchestra.run.smithi016.stdout:(9/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 10 MB/s | 10 MB 00:01 2024-04-25T01:14:18.074 INFO:teuthology.orchestra.run.smithi016.stdout:(10/152): ceph-selinux-19.0.0-2445.gd2c45d97.el 167 kB/s | 25 kB 00:00 2024-04-25T01:14:18.141 INFO:teuthology.orchestra.run.smithi016.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 3.3 MB/s | 222 kB 00:00 2024-04-25T01:14:18.208 INFO:teuthology.orchestra.run.smithi016.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 405 kB/s | 27 kB 00:00 2024-04-25T01:14:18.283 INFO:teuthology.orchestra.run.smithi016.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 423 kB/s | 32 kB 00:00 2024-04-25T01:14:18.417 INFO:teuthology.orchestra.run.smithi016.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 5.5 MB/s | 745 kB 00:00 2024-04-25T01:14:18.492 INFO:teuthology.orchestra.run.smithi016.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.1 MB/s | 163 kB 00:00 2024-04-25T01:14:18.534 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-25T01:14:18.559 INFO:teuthology.orchestra.run.smithi016.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.9 MB/s | 127 kB 00:00 2024-04-25T01:14:18.574 INFO:teuthology.orchestra.run.smithi038.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-25T01:14:18.643 INFO:teuthology.orchestra.run.smithi016.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 5.9 MB/s | 501 kB 00:00 2024-04-25T01:14:19.177 INFO:teuthology.orchestra.run.smithi016.stdout:(18/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 7.2 MB/s | 17 MB 00:02 2024-04-25T01:14:19.269 INFO:teuthology.orchestra.run.smithi016.stdout:(19/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 8.5 MB/s | 5.3 MB 00:00 2024-04-25T01:14:19.274 INFO:teuthology.orchestra.run.smithi038.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 890 kB 00:00 2024-04-25T01:14:19.285 INFO:teuthology.orchestra.run.smithi016.stdout:(20/152): python3-ceph-argparse-19.0.0-2445.gd2 416 kB/s | 45 kB 00:00 2024-04-25T01:14:19.336 INFO:teuthology.orchestra.run.smithi016.stdout:(21/152): python3-ceph-common-19.0.0-2445.gd2c4 1.9 MB/s | 133 kB 00:00 2024-04-25T01:14:19.361 INFO:teuthology.orchestra.run.smithi016.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.1 MB/s | 162 kB 00:00 2024-04-25T01:14:19.375 INFO:teuthology.orchestra.run.smithi038.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.4 MB/s | 145 kB 00:00 2024-04-25T01:14:19.403 INFO:teuthology.orchestra.run.smithi016.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.7 MB/s | 321 kB 00:00 2024-04-25T01:14:19.437 INFO:teuthology.orchestra.run.smithi016.stdout:(24/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.9 MB/s | 299 kB 00:00 2024-04-25T01:14:19.462 INFO:teuthology.orchestra.run.smithi016.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.7 MB/s | 99 kB 00:00 2024-04-25T01:14:19.504 INFO:teuthology.orchestra.run.smithi016.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-25T01:14:19.604 INFO:teuthology.orchestra.run.smithi016.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-25T01:14:19.687 INFO:teuthology.orchestra.run.smithi016.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 381 kB/s | 32 kB 00:00 2024-04-25T01:14:19.779 INFO:teuthology.orchestra.run.smithi016.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.5 MB/s | 141 kB 00:00 2024-04-25T01:14:19.830 INFO:teuthology.orchestra.run.smithi016.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 8.4 MB/s | 3.1 MB 00:00 2024-04-25T01:14:20.197 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T01:14:20.316 INFO:teuthology.orchestra.run.smithi038.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 2.5 MB/s | 2.4 MB 00:00 2024-04-25T01:14:20.430 INFO:teuthology.orchestra.run.smithi016.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 5.4 MB/s | 3.5 MB 00:00 2024-04-25T01:14:20.530 INFO:teuthology.orchestra.run.smithi016.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.4 MB/s | 249 kB 00:00 2024-04-25T01:14:20.614 INFO:teuthology.orchestra.run.smithi016.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 593 kB/s | 49 kB 00:00 2024-04-25T01:14:20.697 INFO:teuthology.orchestra.run.smithi016.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 184 kB/s | 15 kB 00:00 2024-04-25T01:14:20.742 INFO:teuthology.orchestra.run.smithi038.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-04-25T01:14:20.798 INFO:teuthology.orchestra.run.smithi016.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.6 MB/s | 264 kB 00:00 2024-04-25T01:14:20.890 INFO:teuthology.orchestra.run.smithi016.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-04-25T01:14:20.965 INFO:teuthology.orchestra.run.smithi016.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 4.5 MB/s | 762 kB 00:00 2024-04-25T01:14:20.990 INFO:teuthology.orchestra.run.smithi016.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 182 kB/s | 18 kB 00:00 2024-04-25T01:14:21.124 INFO:teuthology.orchestra.run.smithi016.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 506 kB/s | 80 kB 00:00 2024-04-25T01:14:21.208 INFO:teuthology.orchestra.run.smithi016.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-25T01:14:21.233 INFO:teuthology.orchestra.run.smithi016.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-04-25T01:14:21.259 INFO:teuthology.orchestra.run.smithi016.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-25T01:14:21.317 INFO:teuthology.orchestra.run.smithi016.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-25T01:14:21.334 INFO:teuthology.orchestra.run.smithi016.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-25T01:14:21.368 INFO:teuthology.orchestra.run.smithi016.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T01:14:21.393 INFO:teuthology.orchestra.run.smithi016.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T01:14:21.418 INFO:teuthology.orchestra.run.smithi016.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T01:14:21.458 INFO:teuthology.orchestra.run.smithi038.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 6.5 MB/s | 4.7 MB 00:00 2024-04-25T01:14:21.477 INFO:teuthology.orchestra.run.smithi016.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 8.0 MB/s | 689 kB 00:00 2024-04-25T01:14:21.511 INFO:teuthology.orchestra.run.smithi016.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-04-25T01:14:21.536 INFO:teuthology.orchestra.run.smithi016.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T01:14:21.553 INFO:teuthology.orchestra.run.smithi016.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 513 kB/s | 21 kB 00:00 2024-04-25T01:14:21.587 INFO:teuthology.orchestra.run.smithi016.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T01:14:21.604 INFO:teuthology.orchestra.run.smithi016.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 650 kB/s | 33 kB 00:00 2024-04-25T01:14:21.645 INFO:teuthology.orchestra.run.smithi016.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 441 kB/s | 18 kB 00:00 2024-04-25T01:14:21.704 INFO:teuthology.orchestra.run.smithi016.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-25T01:14:21.721 INFO:teuthology.orchestra.run.smithi016.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-04-25T01:14:21.755 INFO:teuthology.orchestra.run.smithi016.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T01:14:21.772 INFO:teuthology.orchestra.run.smithi016.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-04-25T01:14:21.800 INFO:teuthology.orchestra.run.smithi038.stdout:(7/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-25T01:14:21.814 INFO:teuthology.orchestra.run.smithi016.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T01:14:21.847 INFO:teuthology.orchestra.run.smithi016.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-04-25T01:14:21.873 INFO:teuthology.orchestra.run.smithi016.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-04-25T01:14:21.898 INFO:teuthology.orchestra.run.smithi016.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-04-25T01:14:21.923 INFO:teuthology.orchestra.run.smithi016.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T01:14:21.948 INFO:teuthology.orchestra.run.smithi016.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T01:14:21.974 INFO:teuthology.orchestra.run.smithi016.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-25T01:14:22.433 INFO:teuthology.orchestra.run.smithi016.stdout:(66/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 11 MB/s | 48 MB 00:04 2024-04-25T01:14:22.533 INFO:teuthology.orchestra.run.smithi016.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-04-25T01:14:22.600 INFO:teuthology.orchestra.run.smithi016.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 13 MB/s | 911 kB 00:00 2024-04-25T01:14:22.658 INFO:teuthology.orchestra.run.smithi016.stdout:(69/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.7 MB/s | 4.7 MB 00:00 2024-04-25T01:14:22.684 INFO:teuthology.orchestra.run.smithi016.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-25T01:14:22.751 INFO:teuthology.orchestra.run.smithi016.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-25T01:14:22.751 INFO:teuthology.orchestra.run.smithi038.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 13 MB/s | 17 MB 00:01 2024-04-25T01:14:22.801 INFO:teuthology.orchestra.run.smithi016.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-25T01:14:22.809 INFO:teuthology.orchestra.run.smithi038.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 433 kB/s | 25 kB 00:00 2024-04-25T01:14:22.843 INFO:teuthology.orchestra.run.smithi016.stdout:(73/152): postgresql-13.14-1.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-04-25T01:14:22.893 INFO:teuthology.orchestra.run.smithi016.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-25T01:14:22.946 INFO:teuthology.orchestra.run.smithi016.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-25T01:14:23.027 INFO:teuthology.orchestra.run.smithi016.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-04-25T01:14:23.044 INFO:teuthology.orchestra.run.smithi016.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-25T01:14:23.069 INFO:teuthology.orchestra.run.smithi016.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 461 kB/s | 19 kB 00:00 2024-04-25T01:14:23.094 INFO:teuthology.orchestra.run.smithi016.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T01:14:23.120 INFO:teuthology.orchestra.run.smithi016.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-04-25T01:14:23.178 INFO:teuthology.orchestra.run.smithi016.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-25T01:14:23.193 INFO:teuthology.orchestra.run.smithi038.stdout:(10/152): ceph-common-19.0.0-2445.gd2c45d97.el9 4.3 MB/s | 21 MB 00:04 2024-04-25T01:14:23.260 INFO:teuthology.orchestra.run.smithi038.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 3.3 MB/s | 222 kB 00:00 2024-04-25T01:14:23.287 INFO:teuthology.orchestra.run.smithi016.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-25T01:14:23.329 INFO:teuthology.orchestra.run.smithi016.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-25T01:14:23.343 INFO:teuthology.orchestra.run.smithi038.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 325 kB/s | 27 kB 00:00 2024-04-25T01:14:23.427 INFO:teuthology.orchestra.run.smithi038.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 382 kB/s | 32 kB 00:00 2024-04-25T01:14:23.462 INFO:teuthology.orchestra.run.smithi016.stdout:(84/152): postgresql-server-13.14-1.el9.x86_64. 7.2 MB/s | 5.8 MB 00:00 2024-04-25T01:14:23.488 INFO:teuthology.orchestra.run.smithi016.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 266 kB/s | 42 kB 00:00 2024-04-25T01:14:23.513 INFO:teuthology.orchestra.run.smithi016.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-25T01:14:23.538 INFO:teuthology.orchestra.run.smithi016.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T01:14:23.564 INFO:teuthology.orchestra.run.smithi016.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T01:14:23.577 INFO:teuthology.orchestra.run.smithi038.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.9 MB/s | 745 kB 00:00 2024-04-25T01:14:23.589 INFO:teuthology.orchestra.run.smithi016.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T01:14:23.614 INFO:teuthology.orchestra.run.smithi016.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T01:14:23.678 INFO:teuthology.orchestra.run.smithi038.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.6 MB/s | 163 kB 00:00 2024-04-25T01:14:23.689 INFO:teuthology.orchestra.run.smithi016.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.6 MB/s | 5.1 MB 00:00 2024-04-25T01:14:23.714 INFO:teuthology.orchestra.run.smithi016.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-25T01:14:23.740 INFO:teuthology.orchestra.run.smithi016.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-04-25T01:14:23.765 INFO:teuthology.orchestra.run.smithi016.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-25T01:14:23.770 INFO:teuthology.orchestra.run.smithi038.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.4 MB/s | 127 kB 00:00 2024-04-25T01:14:23.790 INFO:teuthology.orchestra.run.smithi016.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-04-25T01:14:23.816 INFO:teuthology.orchestra.run.smithi016.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-25T01:14:23.841 INFO:teuthology.orchestra.run.smithi016.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-25T01:14:23.869 INFO:teuthology.orchestra.run.smithi016.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T01:14:23.878 INFO:teuthology.orchestra.run.smithi038.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 4.5 MB/s | 501 kB 00:00 2024-04-25T01:14:23.894 INFO:teuthology.orchestra.run.smithi016.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T01:14:23.919 INFO:teuthology.orchestra.run.smithi016.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T01:14:23.944 INFO:teuthology.orchestra.run.smithi016.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-25T01:14:24.003 INFO:teuthology.orchestra.run.smithi016.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-04-25T01:14:24.253 INFO:teuthology.orchestra.run.smithi016.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-25T01:14:24.320 INFO:teuthology.orchestra.run.smithi016.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-04-25T01:14:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-25T01:14:24.371 INFO:teuthology.orchestra.run.smithi016.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-04-25T01:14:24.396 INFO:teuthology.orchestra.run.smithi016.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 257 kB/s | 19 kB 00:00 2024-04-25T01:14:24.421 INFO:teuthology.orchestra.run.smithi016.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-04-25T01:14:24.446 INFO:teuthology.orchestra.run.smithi016.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 966 kB/s | 49 kB 00:00 2024-04-25T01:14:24.471 INFO:teuthology.orchestra.run.smithi016.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T01:14:24.505 INFO:teuthology.orchestra.run.smithi016.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-25T01:14:24.514 INFO:teuthology.orchestra.run.smithi038.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 8.4 MB/s | 5.3 MB 00:00 2024-04-25T01:14:24.547 INFO:teuthology.orchestra.run.smithi016.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-04-25T01:14:24.598 INFO:teuthology.orchestra.run.smithi038.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 544 kB/s | 45 kB 00:00 2024-04-25T01:14:24.614 INFO:teuthology.orchestra.run.smithi016.stdout:(113/152): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T01:14:24.631 INFO:teuthology.orchestra.run.smithi016.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-04-25T01:14:24.648 INFO:teuthology.orchestra.run.smithi016.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 962 kB/s | 32 kB 00:00 2024-04-25T01:14:24.665 INFO:teuthology.orchestra.run.smithi016.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 448 kB/s | 15 kB 00:00 2024-04-25T01:14:24.690 INFO:teuthology.orchestra.run.smithi038.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.4 MB/s | 133 kB 00:00 2024-04-25T01:14:24.699 INFO:teuthology.orchestra.run.smithi016.stdout:(117/152): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-04-25T01:14:24.725 INFO:teuthology.orchestra.run.smithi016.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-25T01:14:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T01:14:24.775 INFO:teuthology.orchestra.run.smithi016.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-04-25T01:14:24.783 INFO:teuthology.orchestra.run.smithi038.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-25T01:14:24.808 INFO:teuthology.orchestra.run.smithi016.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-25T01:14:24.833 INFO:teuthology.orchestra.run.smithi016.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-25T01:14:24.858 INFO:teuthology.orchestra.run.smithi016.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-25T01:14:24.884 INFO:teuthology.orchestra.run.smithi016.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-04-25T01:14:24.886 INFO:teuthology.orchestra.run.smithi038.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.1 MB/s | 321 kB 00:00 2024-04-25T01:14:24.909 INFO:teuthology.orchestra.run.smithi016.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-25T01:14:24.934 INFO:teuthology.orchestra.run.smithi016.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-25T01:14:24.959 INFO:teuthology.orchestra.run.smithi016.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-04-25T01:14:24.974 INFO:teuthology.orchestra.run.smithi038.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.2 MB/s | 299 kB 00:00 2024-04-25T01:14:24.987 INFO:teuthology.orchestra.run.smithi016.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 389 kB/s | 20 kB 00:00 2024-04-25T01:14:25.019 INFO:teuthology.orchestra.run.smithi016.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-04-25T01:14:25.044 INFO:teuthology.orchestra.run.smithi016.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-04-25T01:14:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.2 MB/s | 99 kB 00:00 2024-04-25T01:14:25.069 INFO:teuthology.orchestra.run.smithi016.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-25T01:14:25.095 INFO:teuthology.orchestra.run.smithi016.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-25T01:14:25.120 INFO:teuthology.orchestra.run.smithi016.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-25T01:14:25.145 INFO:teuthology.orchestra.run.smithi016.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-25T01:14:25.168 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-25T01:14:25.187 INFO:teuthology.orchestra.run.smithi016.stdout:(135/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.9 MB/s | 2.0 MB 00:00 2024-04-25T01:14:25.213 INFO:teuthology.orchestra.run.smithi016.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-25T01:14:25.237 INFO:teuthology.orchestra.run.smithi016.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 767 kB/s | 90 kB 00:00 2024-04-25T01:14:25.263 INFO:teuthology.orchestra.run.smithi016.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-25T01:14:25.288 INFO:teuthology.orchestra.run.smithi016.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T01:14:25.313 INFO:teuthology.orchestra.run.smithi016.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 718 kB/s | 36 kB 00:00 2024-04-25T01:14:25.355 INFO:teuthology.orchestra.run.smithi016.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-25T01:14:25.381 INFO:teuthology.orchestra.run.smithi016.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-25T01:14:25.406 INFO:teuthology.orchestra.run.smithi016.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T01:14:25.431 INFO:teuthology.orchestra.run.smithi016.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-25T01:14:25.456 INFO:teuthology.orchestra.run.smithi016.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-04-25T01:14:25.481 INFO:teuthology.orchestra.run.smithi016.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T01:14:25.492 INFO:teuthology.orchestra.run.smithi038.stdout:(25/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 18 MB/s | 48 MB 00:02 2024-04-25T01:14:25.507 INFO:teuthology.orchestra.run.smithi016.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-25T01:14:25.517 INFO:teuthology.orchestra.run.smithi038.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 186 kB/s | 85 kB 00:00 2024-04-25T01:14:25.549 INFO:teuthology.orchestra.run.smithi016.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-25T01:14:25.592 INFO:teuthology.orchestra.run.smithi038.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-25T01:14:25.692 INFO:teuthology.orchestra.run.smithi038.stdout:(28/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 15 MB/s | 3.1 MB 00:00 2024-04-25T01:14:25.699 INFO:teuthology.orchestra.run.smithi016.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 1.3 MB 00:00 2024-04-25T01:14:25.718 INFO:teuthology.orchestra.run.smithi038.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 253 kB/s | 32 kB 00:00 2024-04-25T01:14:25.743 INFO:teuthology.orchestra.run.smithi038.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.7 MB/s | 141 kB 00:00 2024-04-25T01:14:25.816 INFO:teuthology.orchestra.run.smithi016.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-04-25T01:14:25.933 INFO:teuthology.orchestra.run.smithi016.stdout:(151/152): librados2-19.0.0-2445.gd2c45d97.el9. 7.5 MB/s | 3.4 MB 00:00 2024-04-25T01:14:26.177 INFO:teuthology.orchestra.run.smithi038.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.7 MB/s | 3.5 MB 00:00 2024-04-25T01:14:26.269 INFO:teuthology.orchestra.run.smithi038.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.7 MB/s | 249 kB 00:00 2024-04-25T01:14:26.308 INFO:teuthology.orchestra.run.smithi016.stdout:(152/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.9 MB/s | 3.1 MB 00:00 2024-04-25T01:14:26.311 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:14:26.311 INFO:teuthology.orchestra.run.smithi016.stdout:Total 20 MB/s | 220 MB 00:11 2024-04-25T01:14:26.386 INFO:teuthology.orchestra.run.smithi038.stdout:(33/152): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-25T01:14:26.411 INFO:teuthology.orchestra.run.smithi038.stdout:(34/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 347 kB/s | 49 kB 00:00 2024-04-25T01:14:26.436 INFO:teuthology.orchestra.run.smithi038.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 306 kB/s | 15 kB 00:00 2024-04-25T01:14:26.462 INFO:teuthology.orchestra.run.smithi038.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 5.1 MB/s | 264 kB 00:00 2024-04-25T01:14:26.504 INFO:teuthology.orchestra.run.smithi038.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 436 kB/s | 18 kB 00:00 2024-04-25T01:14:26.537 INFO:teuthology.orchestra.run.smithi038.stdout:(38/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 7.4 MB/s | 762 kB 00:00 2024-04-25T01:14:26.704 INFO:teuthology.orchestra.run.smithi038.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 401 kB/s | 80 kB 00:00 2024-04-25T01:14:26.804 INFO:teuthology.orchestra.run.smithi038.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-04-25T01:14:26.913 INFO:teuthology.orchestra.run.smithi038.stdout:(41/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 2.0 MB/s | 10 MB 00:05 2024-04-25T01:14:26.939 INFO:teuthology.orchestra.run.smithi038.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 253 kB 00:00 2024-04-25T01:14:26.964 INFO:teuthology.orchestra.run.smithi038.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-04-25T01:14:26.997 INFO:teuthology.orchestra.run.smithi038.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T01:14:27.023 INFO:teuthology.orchestra.run.smithi038.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-25T01:14:27.056 INFO:teuthology.orchestra.run.smithi038.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T01:14:27.090 INFO:teuthology.orchestra.run.smithi038.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-25T01:14:27.232 INFO:teuthology.orchestra.run.smithi038.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 3.8 MB/s | 1.2 MB 00:00 2024-04-25T01:14:27.312 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:14:27.312 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:14:27.323 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:14:27.324 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:14:27.324 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:14:27.325 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:14:27.327 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:14:27.329 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:14:27.330 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:14:27.390 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-25T01:14:27.399 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:14:27.400 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:14:27.401 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:14:27.402 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:14:27.403 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:14:27.404 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:14:27.405 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:14:27.406 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:14:27.407 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:14:27.408 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:14:27.409 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:14:27.410 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:14:27.411 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout:Install 148 Packages 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 4 Packages 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:14:27.412 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 220 M 2024-04-25T01:14:27.413 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-04-25T01:14:27.474 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-25T01:14:27.635 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-25T01:14:27.635 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-25T01:14:27.682 INFO:teuthology.orchestra.run.smithi038.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 237 kB/s | 107 kB 00:00 2024-04-25T01:14:27.783 INFO:teuthology.orchestra.run.smithi038.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-04-25T01:14:27.816 INFO:teuthology.orchestra.run.smithi038.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 907 kB/s | 689 kB 00:00 2024-04-25T01:14:27.917 INFO:teuthology.orchestra.run.smithi038.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-04-25T01:14:27.984 INFO:teuthology.orchestra.run.smithi038.stdout:(53/152): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-04-25T01:14:28.092 INFO:teuthology.orchestra.run.smithi038.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-04-25T01:14:28.126 INFO:teuthology.orchestra.run.smithi038.stdout:(55/152): flex-2.6.4-9.el9.x86_64.rpm 915 kB/s | 313 kB 00:00 2024-04-25T01:14:28.201 INFO:teuthology.orchestra.run.smithi038.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 102 kB/s | 11 kB 00:00 2024-04-25T01:14:28.226 INFO:teuthology.orchestra.run.smithi038.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-25T01:14:28.302 INFO:teuthology.orchestra.run.smithi038.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-04-25T01:14:28.312 INFO:teuthology.orchestra.run.smithi156.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-25T01:14:28.469 INFO:teuthology.orchestra.run.smithi038.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.4 MB/s | 3.0 MB 00:00 2024-04-25T01:14:28.494 INFO:teuthology.orchestra.run.smithi038.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-04-25T01:14:28.529 INFO:teuthology.orchestra.run.smithi156.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 4.0 MB/s | 890 kB 00:00 2024-04-25T01:14:28.569 INFO:teuthology.orchestra.run.smithi038.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-25T01:14:28.595 INFO:teuthology.orchestra.run.smithi156.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.1 MB/s | 145 kB 00:00 2024-04-25T01:14:28.603 INFO:teuthology.orchestra.run.smithi038.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 1.9 MB/s | 582 kB 00:00 2024-04-25T01:14:28.628 INFO:teuthology.orchestra.run.smithi038.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 354 kB/s | 47 kB 00:00 2024-04-25T01:14:28.679 INFO:teuthology.orchestra.run.smithi038.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-25T01:14:28.712 INFO:teuthology.orchestra.run.smithi038.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-25T01:14:28.737 INFO:teuthology.orchestra.run.smithi038.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 323 kB/s | 35 kB 00:00 2024-04-25T01:14:28.929 INFO:teuthology.orchestra.run.smithi156.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 6.8 MB/s | 5.4 MB 00:00 2024-04-25T01:14:28.987 INFO:teuthology.orchestra.run.smithi156.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 6.0 MB/s | 2.4 MB 00:00 2024-04-25T01:14:29.021 INFO:teuthology.orchestra.run.smithi038.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-25T01:14:29.121 INFO:teuthology.orchestra.run.smithi156.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-04-25T01:14:29.138 INFO:teuthology.orchestra.run.smithi038.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-04-25T01:14:29.238 INFO:teuthology.orchestra.run.smithi038.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-25T01:14:29.501 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-25T01:14:29.502 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-25T01:14:29.697 INFO:teuthology.orchestra.run.smithi038.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-04-25T01:14:29.963 INFO:teuthology.orchestra.run.smithi156.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 4.8 MB/s | 4.7 MB 00:00 2024-04-25T01:14:30.523 INFO:teuthology.orchestra.run.smithi156.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 8.8 MB/s | 21 MB 00:02 2024-04-25T01:14:30.688 INFO:teuthology.orchestra.run.smithi156.stdout:(9/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 11 MB/s | 17 MB 00:01 2024-04-25T01:14:30.705 INFO:teuthology.orchestra.run.smithi156.stdout:(10/152): ceph-selinux-19.0.0-2445.gd2c45d97.el 137 kB/s | 25 kB 00:00 2024-04-25T01:14:30.755 INFO:teuthology.orchestra.run.smithi156.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 4.4 MB/s | 222 kB 00:00 2024-04-25T01:14:30.814 INFO:teuthology.orchestra.run.smithi156.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 465 kB/s | 27 kB 00:00 2024-04-25T01:14:30.873 INFO:teuthology.orchestra.run.smithi156.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 540 kB/s | 32 kB 00:00 2024-04-25T01:14:31.023 INFO:teuthology.orchestra.run.smithi156.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.9 MB/s | 745 kB 00:00 2024-04-25T01:14:31.107 INFO:teuthology.orchestra.run.smithi156.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 163 kB 00:00 2024-04-25T01:14:31.147 INFO:teuthology.orchestra.run.smithi038.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 713 kB/s | 1.0 MB 00:01 2024-04-25T01:14:31.182 INFO:teuthology.orchestra.run.smithi156.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.7 MB/s | 127 kB 00:00 2024-04-25T01:14:31.274 INFO:teuthology.orchestra.run.smithi156.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 5.3 MB/s | 501 kB 00:00 2024-04-25T01:14:31.347 INFO:teuthology.orchestra.run.smithi038.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 437 kB/s | 87 kB 00:00 2024-04-25T01:14:31.782 INFO:teuthology.orchestra.run.smithi156.stdout:(18/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 5.6 MB/s | 10 MB 00:01 2024-04-25T01:14:31.858 INFO:teuthology.orchestra.run.smithi156.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 602 kB/s | 45 kB 00:00 2024-04-25T01:14:31.941 INFO:teuthology.orchestra.run.smithi156.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.6 MB/s | 133 kB 00:00 2024-04-25T01:14:32.017 INFO:teuthology.orchestra.run.smithi156.stdout:(21/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 7.2 MB/s | 5.3 MB 00:00 2024-04-25T01:14:32.042 INFO:teuthology.orchestra.run.smithi156.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.6 MB/s | 162 kB 00:00 2024-04-25T01:14:32.084 INFO:teuthology.orchestra.run.smithi156.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.7 MB/s | 321 kB 00:00 2024-04-25T01:14:32.126 INFO:teuthology.orchestra.run.smithi156.stdout:(24/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-25T01:14:32.156 INFO:teuthology.orchestra.run.smithi156.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.5 MB/s | 99 kB 00:00 2024-04-25T01:14:32.193 INFO:teuthology.orchestra.run.smithi156.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-25T01:14:32.280 INFO:teuthology.orchestra.run.smithi156.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-25T01:14:32.361 INFO:teuthology.orchestra.run.smithi156.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 381 kB/s | 32 kB 00:00 2024-04-25T01:14:32.453 INFO:teuthology.orchestra.run.smithi156.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.5 MB/s | 141 kB 00:00 2024-04-25T01:14:32.570 INFO:teuthology.orchestra.run.smithi156.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 7.4 MB/s | 3.1 MB 00:00 2024-04-25T01:14:33.089 INFO:teuthology.orchestra.run.smithi038.stdout:(73/152): openblas-openmp-0.3.21-2.el9.x86_64.r 1.1 MB/s | 4.7 MB 00:04 2024-04-25T01:14:33.237 INFO:teuthology.orchestra.run.smithi156.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 4.5 MB/s | 3.5 MB 00:00 2024-04-25T01:14:33.337 INFO:teuthology.orchestra.run.smithi156.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.4 MB/s | 249 kB 00:00 2024-04-25T01:14:33.472 INFO:teuthology.orchestra.run.smithi038.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 637 kB/s | 244 kB 00:00 2024-04-25T01:14:33.754 INFO:teuthology.orchestra.run.smithi156.stdout:(33/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 16 MB/s | 48 MB 00:03 2024-04-25T01:14:33.779 INFO:teuthology.orchestra.run.smithi156.stdout:(34/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 111 kB/s | 49 kB 00:00 2024-04-25T01:14:33.796 INFO:teuthology.orchestra.run.smithi156.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 369 kB/s | 15 kB 00:00 2024-04-25T01:14:33.823 INFO:teuthology.orchestra.run.smithi038.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 708 kB/s | 248 kB 00:00 2024-04-25T01:14:33.888 INFO:teuthology.orchestra.run.smithi156.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-04-25T01:14:33.914 INFO:teuthology.orchestra.run.smithi156.stdout:(37/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 1.9 MB/s | 264 kB 00:00 2024-04-25T01:14:33.923 INFO:teuthology.orchestra.run.smithi038.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-25T01:14:33.930 INFO:teuthology.orchestra.run.smithi156.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 439 kB/s | 18 kB 00:00 2024-04-25T01:14:33.966 INFO:teuthology.orchestra.run.smithi156.stdout:(39/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 4.4 MB/s | 762 kB 00:00 2024-04-25T01:14:34.023 INFO:teuthology.orchestra.run.smithi038.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 192 kB/s | 19 kB 00:00 2024-04-25T01:14:34.141 INFO:teuthology.orchestra.run.smithi156.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 353 kB/s | 80 kB 00:00 2024-04-25T01:14:34.208 INFO:teuthology.orchestra.run.smithi156.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 807 kB/s | 195 kB 00:00 2024-04-25T01:14:34.317 INFO:teuthology.orchestra.run.smithi156.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-04-25T01:14:34.382 INFO:teuthology.orchestra.run.smithi038.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 481 kB/s | 172 kB 00:00 2024-04-25T01:14:34.409 INFO:teuthology.orchestra.run.smithi156.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 948 kB/s | 253 kB 00:00 2024-04-25T01:14:34.482 INFO:teuthology.orchestra.run.smithi038.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 348 kB/s | 35 kB 00:00 2024-04-25T01:14:34.559 INFO:teuthology.orchestra.run.smithi156.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 441 kB/s | 106 kB 00:00 2024-04-25T01:14:34.659 INFO:teuthology.orchestra.run.smithi156.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 540 kB/s | 135 kB 00:00 2024-04-25T01:14:34.784 INFO:teuthology.orchestra.run.smithi156.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 554 kB/s | 125 kB 00:00 2024-04-25T01:14:34.860 INFO:teuthology.orchestra.run.smithi156.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 1.9 MB/s | 1.2 MB 00:00 2024-04-25T01:14:34.960 INFO:teuthology.orchestra.run.smithi156.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 714 kB/s | 215 kB 00:00 2024-04-25T01:14:35.116 INFO:teuthology.orchestra.run.smithi038.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 1.0 MB/s | 5.8 MB 00:05 2024-04-25T01:14:35.177 INFO:teuthology.orchestra.run.smithi156.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 493 kB/s | 107 kB 00:00 2024-04-25T01:14:35.225 INFO:teuthology.orchestra.run.smithi156.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 689 kB 00:00 2024-04-25T01:14:35.332 INFO:teuthology.orchestra.run.smithi156.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 943 kB 00:00 2024-04-25T01:14:35.361 INFO:teuthology.orchestra.run.smithi156.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 117 kB/s | 21 kB 00:00 2024-04-25T01:14:35.491 INFO:teuthology.orchestra.run.smithi038.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 668 kB/s | 250 kB 00:00 2024-04-25T01:14:35.511 INFO:teuthology.orchestra.run.smithi156.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 183 kB/s | 33 kB 00:00 2024-04-25T01:14:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-25T01:14:35.604 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:14:35.643 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:14:35.645 INFO:teuthology.orchestra.run.smithi156.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 737 kB/s | 313 kB 00:00 2024-04-25T01:14:35.671 INFO:teuthology.orchestra.run.smithi156.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 115 kB/s | 18 kB 00:00 2024-04-25T01:14:35.697 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:14:35.739 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:14:35.764 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:14:35.796 INFO:teuthology.orchestra.run.smithi156.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 74 kB/s | 11 kB 00:00 2024-04-25T01:14:35.808 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:14:35.912 INFO:teuthology.orchestra.run.smithi038.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 533 kB/s | 222 kB 00:00 2024-04-25T01:14:35.972 INFO:teuthology.orchestra.run.smithi156.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 533 kB/s | 160 kB 00:00 2024-04-25T01:14:36.047 INFO:teuthology.orchestra.run.smithi156.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 194 kB/s | 45 kB 00:00 2024-04-25T01:14:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 371 kB/s | 77 kB 00:00 2024-04-25T01:14:36.155 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:14:36.248 INFO:teuthology.orchestra.run.smithi038.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 362 kB/s | 42 kB 00:00 2024-04-25T01:14:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:36.397 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:36.439 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:36.505 INFO:teuthology.orchestra.run.smithi156.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.2 MB/s | 662 kB 00:00 2024-04-25T01:14:36.524 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:36.530 INFO:teuthology.orchestra.run.smithi156.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 1.1 MB/s | 582 kB 00:00 2024-04-25T01:14:36.556 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:14:36.597 INFO:teuthology.orchestra.run.smithi156.stdout:(61/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.4 MB/s | 3.0 MB 00:01 2024-04-25T01:14:36.639 INFO:teuthology.orchestra.run.smithi156.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-04-25T01:14:36.692 INFO:teuthology.orchestra.run.smithi038.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 588 kB/s | 269 kB 00:00 2024-04-25T01:14:36.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:14:36.731 INFO:teuthology.orchestra.run.smithi156.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 236 kB/s | 47 kB 00:00 2024-04-25T01:14:36.767 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:14:36.794 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:36.864 INFO:teuthology.orchestra.run.smithi156.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 265 kB/s | 35 kB 00:00 2024-04-25T01:14:36.883 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:36.889 INFO:teuthology.orchestra.run.smithi156.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 753 kB/s | 188 kB 00:00 2024-04-25T01:14:36.912 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:36.940 INFO:teuthology.orchestra.run.smithi156.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 853 kB/s | 292 kB 00:00 2024-04-25T01:14:36.958 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:36.967 INFO:teuthology.orchestra.run.smithi038.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 577 kB/s | 159 kB 00:00 2024-04-25T01:14:36.988 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:37.009 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:37.164 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:37.196 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:37.307 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:14:37.413 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:14:37.453 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:14:37.490 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:14:37.559 INFO:teuthology.orchestra.run.smithi038.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 471 kB/s | 279 kB 00:00 2024-04-25T01:14:37.567 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:14:37.605 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:14:37.649 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:14:37.659 INFO:teuthology.orchestra.run.smithi038.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 511 kB/s | 51 kB 00:00 2024-04-25T01:14:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:14:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:14:37.760 INFO:teuthology.orchestra.run.smithi038.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 537 kB/s | 54 kB 00:00 2024-04-25T01:14:37.782 INFO:teuthology.orchestra.run.smithi156.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-25T01:14:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:14:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:14:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:14:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:14:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:14:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:14:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:14:38.129 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:14:38.199 INFO:teuthology.orchestra.run.smithi156.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 2.1 MB/s | 911 kB 00:00 2024-04-25T01:14:38.204 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:14:38.253 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:14:38.299 INFO:teuthology.orchestra.run.smithi156.stdout:(69/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.3 MB/s | 4.7 MB 00:01 2024-04-25T01:14:38.319 INFO:teuthology.orchestra.run.smithi038.stdout:(90/152): python3-babel-2.9.1-2.el9.noarch.rpm 876 kB/s | 6.0 MB 00:06 2024-04-25T01:14:38.331 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:14:38.427 INFO:teuthology.orchestra.run.smithi038.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 385 kB/s | 42 kB 00:00 2024-04-25T01:14:38.457 INFO:teuthology.orchestra.run.smithi156.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.8 MB/s | 4.4 MB 00:01 2024-04-25T01:14:38.477 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:14:38.527 INFO:teuthology.orchestra.run.smithi038.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-04-25T01:14:38.531 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:14:38.533 INFO:teuthology.orchestra.run.smithi156.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 399 kB/s | 133 kB 00:00 2024-04-25T01:14:38.708 INFO:teuthology.orchestra.run.smithi156.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 500 kB/s | 87 kB 00:00 2024-04-25T01:14:38.867 INFO:teuthology.orchestra.run.smithi156.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-04-25T01:14:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 630 kB/s | 221 kB 00:00 2024-04-25T01:14:38.978 INFO:teuthology.orchestra.run.smithi038.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 358 kB/s | 36 kB 00:00 2024-04-25T01:14:39.087 INFO:teuthology.orchestra.run.smithi038.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-04-25T01:14:39.088 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:14:39.125 INFO:teuthology.orchestra.run.smithi156.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 945 kB/s | 244 kB 00:00 2024-04-25T01:14:39.177 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:14:39.212 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:14:39.255 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:14:39.293 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:14:39.336 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:14:39.385 INFO:teuthology.orchestra.run.smithi156.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 961 kB/s | 248 kB 00:00 2024-04-25T01:14:39.462 INFO:teuthology.orchestra.run.smithi038.stdout:(96/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.0 MB/s | 5.1 MB 00:04 2024-04-25T01:14:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:14:39.526 INFO:teuthology.orchestra.run.smithi156.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 181 kB/s | 26 kB 00:00 2024-04-25T01:14:39.554 INFO:teuthology.orchestra.run.smithi038.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 652 kB/s | 305 kB 00:00 2024-04-25T01:14:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:14:39.579 INFO:teuthology.orchestra.run.smithi038.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 493 kB/s | 58 kB 00:00 2024-04-25T01:14:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:14:39.668 INFO:teuthology.orchestra.run.smithi156.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 135 kB/s | 19 kB 00:00 2024-04-25T01:14:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:14:39.738 INFO:teuthology.orchestra.run.smithi038.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 511 kB/s | 94 kB 00:00 2024-04-25T01:14:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:14:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:14:39.780 INFO:teuthology.orchestra.run.smithi038.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 318 kB/s | 64 kB 00:00 2024-04-25T01:14:39.818 INFO:teuthology.orchestra.run.smithi156.stdout:(78/152): postgresql-server-13.14-1.el9.x86_64. 3.8 MB/s | 5.8 MB 00:01 2024-04-25T01:14:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:14:39.838 INFO:teuthology.orchestra.run.smithi038.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 446 kB/s | 45 kB 00:00 2024-04-25T01:14:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:14:39.873 INFO:teuthology.orchestra.run.smithi038.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-04-25T01:14:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:14:39.935 INFO:teuthology.orchestra.run.smithi156.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-04-25T01:14:39.952 INFO:teuthology.orchestra.run.smithi156.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 607 kB/s | 172 kB 00:00 2024-04-25T01:14:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:14:39.977 INFO:teuthology.orchestra.run.smithi156.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 219 kB/s | 35 kB 00:00 2024-04-25T01:14:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:14:40.114 INFO:teuthology.orchestra.run.smithi038.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-04-25T01:14:40.139 INFO:teuthology.orchestra.run.smithi038.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-04-25T01:14:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:14:40.165 INFO:teuthology.orchestra.run.smithi038.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 6.0 MB/s | 309 kB 00:00 2024-04-25T01:14:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:14:40.190 INFO:teuthology.orchestra.run.smithi038.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 393 kB/s | 19 kB 00:00 2024-04-25T01:14:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:14:40.227 INFO:teuthology.orchestra.run.smithi156.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 888 kB/s | 222 kB 00:00 2024-04-25T01:14:40.252 INFO:teuthology.orchestra.run.smithi156.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 834 kB/s | 250 kB 00:00 2024-04-25T01:14:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:14:40.333 INFO:teuthology.orchestra.run.smithi038.stdout:(107/152): libarrow-9.0.0-10.el9.x86_64.rpm 26 MB/s | 4.4 MB 00:00 2024-04-25T01:14:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:14:40.357 INFO:teuthology.orchestra.run.smithi038.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-04-25T01:14:40.383 INFO:teuthology.orchestra.run.smithi038.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-04-25T01:14:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:14:40.408 INFO:teuthology.orchestra.run.smithi038.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T01:14:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:14:40.411 INFO:teuthology.orchestra.run.smithi156.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 421 kB/s | 77 kB 00:00 2024-04-25T01:14:40.425 INFO:teuthology.orchestra.run.smithi038.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-04-25T01:14:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:14:40.436 INFO:teuthology.orchestra.run.smithi156.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 230 kB/s | 42 kB 00:00 2024-04-25T01:14:40.467 INFO:teuthology.orchestra.run.smithi038.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-04-25T01:14:40.492 INFO:teuthology.orchestra.run.smithi038.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-25T01:14:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:14:40.518 INFO:teuthology.orchestra.run.smithi038.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 881 kB/s | 43 kB 00:00 2024-04-25T01:14:40.543 INFO:teuthology.orchestra.run.smithi038.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-04-25T01:14:40.568 INFO:teuthology.orchestra.run.smithi038.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T01:14:40.593 INFO:teuthology.orchestra.run.smithi038.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-25T01:14:40.627 INFO:teuthology.orchestra.run.smithi038.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-25T01:14:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:14:40.652 INFO:teuthology.orchestra.run.smithi038.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-25T01:14:40.678 INFO:teuthology.orchestra.run.smithi038.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 711 kB/s | 35 kB 00:00 2024-04-25T01:14:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:14:40.686 INFO:teuthology.orchestra.run.smithi156.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 978 kB/s | 269 kB 00:00 2024-04-25T01:14:40.703 INFO:teuthology.orchestra.run.smithi038.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-25T01:14:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:14:40.712 INFO:teuthology.orchestra.run.smithi156.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 576 kB/s | 159 kB 00:00 2024-04-25T01:14:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:14:40.762 INFO:teuthology.orchestra.run.smithi038.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 25 MB/s | 2.0 MB 00:00 2024-04-25T01:14:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:14:40.787 INFO:teuthology.orchestra.run.smithi038.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-25T01:14:40.821 INFO:teuthology.orchestra.run.smithi038.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-25T01:14:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:14:40.846 INFO:teuthology.orchestra.run.smithi038.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-04-25T01:14:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:14:40.871 INFO:teuthology.orchestra.run.smithi038.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-25T01:14:40.887 INFO:teuthology.orchestra.run.smithi156.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 292 kB/s | 51 kB 00:00 2024-04-25T01:14:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:14:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-25T01:14:40.922 INFO:teuthology.orchestra.run.smithi038.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-25T01:14:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:14:40.964 INFO:teuthology.orchestra.run.smithi038.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-25T01:14:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:14:40.979 INFO:teuthology.orchestra.run.smithi156.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 955 kB/s | 279 kB 00:00 2024-04-25T01:14:40.989 INFO:teuthology.orchestra.run.smithi038.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 702 kB/s | 46 kB 00:00 2024-04-25T01:14:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:14:41.015 INFO:teuthology.orchestra.run.smithi038.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-25T01:14:41.032 INFO:teuthology.orchestra.run.smithi038.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-04-25T01:14:41.065 INFO:teuthology.orchestra.run.smithi038.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-25T01:14:41.071 INFO:teuthology.orchestra.run.smithi156.stdout:(90/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-04-25T01:14:41.082 INFO:teuthology.orchestra.run.smithi038.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-25T01:14:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:14:41.096 INFO:teuthology.orchestra.run.smithi156.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 257 kB/s | 54 kB 00:00 2024-04-25T01:14:41.116 INFO:teuthology.orchestra.run.smithi038.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-25T01:14:41.133 INFO:teuthology.orchestra.run.smithi038.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-25T01:14:41.158 INFO:teuthology.orchestra.run.smithi038.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 736 kB/s | 31 kB 00:00 2024-04-25T01:14:41.184 INFO:teuthology.orchestra.run.smithi038.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-25T01:14:41.209 INFO:teuthology.orchestra.run.smithi038.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-25T01:14:41.225 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:14:41.234 INFO:teuthology.orchestra.run.smithi038.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-04-25T01:14:41.259 INFO:teuthology.orchestra.run.smithi038.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T01:14:41.271 INFO:teuthology.orchestra.run.smithi156.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 208 kB/s | 42 kB 00:00 2024-04-25T01:14:41.285 INFO:teuthology.orchestra.run.smithi038.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-25T01:14:41.297 INFO:teuthology.orchestra.run.smithi156.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 220 kB/s | 44 kB 00:00 2024-04-25T01:14:41.310 INFO:teuthology.orchestra.run.smithi038.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-25T01:14:41.335 INFO:teuthology.orchestra.run.smithi038.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 8.4 MB/s | 427 kB 00:00 2024-04-25T01:14:41.361 INFO:teuthology.orchestra.run.smithi038.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 404 kB/s | 20 kB 00:00 2024-04-25T01:14:41.361 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:14:41.386 INFO:teuthology.orchestra.run.smithi038.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T01:14:41.411 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:14:41.436 INFO:teuthology.orchestra.run.smithi038.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-25T01:14:41.447 INFO:teuthology.orchestra.run.smithi156.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 239 kB/s | 36 kB 00:00 2024-04-25T01:14:41.524 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:14:41.539 INFO:teuthology.orchestra.run.smithi156.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 826 kB/s | 221 kB 00:00 2024-04-25T01:14:41.580 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:14:41.623 INFO:teuthology.orchestra.run.smithi156.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 270 kB/s | 47 kB 00:00 2024-04-25T01:14:41.703 INFO:teuthology.orchestra.run.smithi038.stdout:(148/152): librados2-19.0.0-2445.gd2c45d97.el9. 11 MB/s | 3.4 MB 00:00 2024-04-25T01:14:41.789 INFO:teuthology.orchestra.run.smithi156.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-04-25T01:14:41.815 INFO:teuthology.orchestra.run.smithi156.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 301 kB/s | 58 kB 00:00 2024-04-25T01:14:41.990 INFO:teuthology.orchestra.run.smithi156.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 467 kB/s | 94 kB 00:00 2024-04-25T01:14:42.007 INFO:teuthology.orchestra.run.smithi156.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 331 kB/s | 64 kB 00:00 2024-04-25T01:14:42.012 INFO:teuthology.orchestra.run.smithi038.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.4 MB/s | 1.3 MB 00:00 2024-04-25T01:14:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 244 kB/s | 45 kB 00:00 2024-04-25T01:14:42.213 INFO:teuthology.orchestra.run.smithi038.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-25T01:14:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-04-25T01:14:42.369 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:14:42.418 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:14:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:14:42.483 INFO:teuthology.orchestra.run.smithi156.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.5 MB/s | 548 kB 00:00 2024-04-25T01:14:42.546 INFO:teuthology.orchestra.run.smithi038.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-04-25T01:14:42.550 INFO:teuthology.orchestra.run.smithi156.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-04-25T01:14:42.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:14:42.593 INFO:teuthology.orchestra.run.smithi156.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-04-25T01:14:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:14:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:14:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:14:42.767 INFO:teuthology.orchestra.run.smithi156.stdout:(106/152): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-04-25T01:14:42.817 INFO:teuthology.orchestra.run.smithi156.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-04-25T01:14:42.945 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:14:42.975 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:14:43.009 INFO:teuthology.orchestra.run.smithi156.stdout:(108/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.0 MB/s | 16 MB 00:02 2024-04-25T01:14:43.026 INFO:teuthology.orchestra.run.smithi156.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-04-25T01:14:43.056 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:14:43.060 INFO:teuthology.orchestra.run.smithi156.stdout:(110/152): protobuf-compiler-3.14.0-13.el9.x86_ 975 kB/s | 863 kB 00:00 2024-04-25T01:14:43.085 INFO:teuthology.orchestra.run.smithi156.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-25T01:14:43.152 INFO:teuthology.orchestra.run.smithi156.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-25T01:14:43.178 INFO:teuthology.orchestra.run.smithi156.stdout:(113/152): libunwind-1.6.2-1.el9.x86_64.rpm 402 kB/s | 67 kB 00:00 2024-04-25T01:14:43.203 INFO:teuthology.orchestra.run.smithi156.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 862 kB/s | 43 kB 00:00 2024-04-25T01:14:43.253 INFO:teuthology.orchestra.run.smithi156.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 301 kB/s | 15 kB 00:00 2024-04-25T01:14:43.278 INFO:teuthology.orchestra.run.smithi156.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-25T01:14:43.312 INFO:teuthology.orchestra.run.smithi156.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T01:14:43.337 INFO:teuthology.orchestra.run.smithi156.stdout:(118/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-04-25T01:14:43.362 INFO:teuthology.orchestra.run.smithi156.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-25T01:14:43.387 INFO:teuthology.orchestra.run.smithi156.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-04-25T01:14:43.413 INFO:teuthology.orchestra.run.smithi156.stdout:(121/152): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-04-25T01:14:43.438 INFO:teuthology.orchestra.run.smithi156.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-25T01:14:43.463 INFO:teuthology.orchestra.run.smithi156.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-25T01:14:43.489 INFO:teuthology.orchestra.run.smithi156.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-25T01:14:43.514 INFO:teuthology.orchestra.run.smithi156.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-25T01:14:43.539 INFO:teuthology.orchestra.run.smithi156.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-25T01:14:43.565 INFO:teuthology.orchestra.run.smithi156.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-04-25T01:14:43.590 INFO:teuthology.orchestra.run.smithi156.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-04-25T01:14:43.624 INFO:teuthology.orchestra.run.smithi156.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-04-25T01:14:43.649 INFO:teuthology.orchestra.run.smithi156.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 791 kB/s | 46 kB 00:00 2024-04-25T01:14:43.674 INFO:teuthology.orchestra.run.smithi156.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-25T01:14:43.725 INFO:teuthology.orchestra.run.smithi156.stdout:(132/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.1 MB/s | 2.0 MB 00:00 2024-04-25T01:14:43.750 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:14:43.750 INFO:teuthology.orchestra.run.smithi156.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-04-25T01:14:43.767 INFO:teuthology.orchestra.run.smithi156.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 394 kB/s | 16 kB 00:00 2024-04-25T01:14:43.792 INFO:teuthology.orchestra.run.smithi156.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-25T01:14:43.809 INFO:teuthology.orchestra.run.smithi156.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 675 kB/s | 28 kB 00:00 2024-04-25T01:14:43.835 INFO:teuthology.orchestra.run.smithi156.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-25T01:14:43.851 INFO:teuthology.orchestra.run.smithi156.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-04-25T01:14:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:14:43.877 INFO:teuthology.orchestra.run.smithi156.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-04-25T01:14:43.893 INFO:teuthology.orchestra.run.smithi156.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 856 kB/s | 36 kB 00:00 2024-04-25T01:14:43.919 INFO:teuthology.orchestra.run.smithi156.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 704 kB/s | 59 kB 00:00 2024-04-25T01:14:43.936 INFO:teuthology.orchestra.run.smithi156.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 869 kB/s | 51 kB 00:00 2024-04-25T01:14:43.961 INFO:teuthology.orchestra.run.smithi156.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-04-25T01:14:43.986 INFO:teuthology.orchestra.run.smithi156.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-25T01:14:44.011 INFO:teuthology.orchestra.run.smithi156.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-25T01:14:44.037 INFO:teuthology.orchestra.run.smithi156.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-25T01:14:44.062 INFO:teuthology.orchestra.run.smithi156.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T01:14:44.104 INFO:teuthology.orchestra.run.smithi156.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-25T01:14:44.371 INFO:teuthology.orchestra.run.smithi156.stdout:(149/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 10 MB/s | 3.1 MB 00:00 2024-04-25T01:14:44.471 INFO:teuthology.orchestra.run.smithi156.stdout:(150/152): librados2-19.0.0-2445.gd2c45d97.el9. 7.8 MB/s | 3.4 MB 00:00 2024-04-25T01:14:44.671 INFO:teuthology.orchestra.run.smithi156.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.4 MB/s | 1.3 MB 00:00 2024-04-25T01:14:44.913 INFO:teuthology.orchestra.run.smithi156.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.4 MB/s | 2.4 MB 00:00 2024-04-25T01:14:44.917 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:14:44.917 INFO:teuthology.orchestra.run.smithi156.stdout:Total 13 MB/s | 220 MB 00:17 2024-04-25T01:14:45.191 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:14:45.271 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:14:45.299 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:14:45.332 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:14:45.661 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:14:45.703 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:14:45.979 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:14:46.022 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-04-25T01:14:46.042 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:14:46.078 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:14:46.166 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-04-25T01:14:46.166 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-04-25T01:14:46.575 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:14:46.606 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:14:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:14:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:14:48.010 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-04-25T01:14:48.010 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-04-25T01:14:49.208 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:49.238 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:49.808 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:49.858 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:14:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:14:49.950 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:14:49.950 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:14:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:14:49.984 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:14:54.274 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-04-25T01:14:54.408 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:14:54.590 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:14:54.651 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:14:54.685 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:14:54.710 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:14:54.753 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:14:55.086 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:14:55.186 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:55.308 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:55.337 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:55.428 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:55.458 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:14:55.609 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:14:55.672 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:14:55.700 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:55.788 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:55.817 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:55.880 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:55.902 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:55.928 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:56.117 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:56.151 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:14:56.355 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:14:56.384 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:14:56.419 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:14:56.484 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:14:56.518 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:14:56.561 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:14:56.594 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:14:56.622 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:14:56.690 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:14:56.743 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:14:56.774 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:14:56.810 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:14:56.886 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:14:56.920 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:14:56.956 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:14:56.984 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:14:57.054 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:14:57.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:14:57.174 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:14:57.308 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:14:57.360 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:14:57.892 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:14:57.981 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:14:58.009 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:14:58.043 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:14:58.090 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:14:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:14:58.286 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:14:58.335 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:14:58.390 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:14:58.462 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:14:58.494 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:14:58.530 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:14:58.577 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:14:58.621 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:14:58.665 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:14:58.706 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:14:58.844 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:14:58.887 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:14:58.917 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:14:58.959 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:14:59.052 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:14:59.083 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:14:59.115 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:14:59.141 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:14:59.165 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:14:59.240 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:14:59.362 INFO:teuthology.orchestra.run.smithi038.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 770 kB/s | 16 MB 00:21 2024-04-25T01:14:59.367 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:14:59.367 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.3 MB/s | 220 MB 00:41 2024-04-25T01:14:59.367 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:14:59.398 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:14:59.426 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:14:59.460 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:14:59.499 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:14:59.531 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:14:59.629 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:14:59.696 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:14:59.729 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:14:59.771 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:14:59.805 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:14:59.893 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:14:59.936 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:14:59.966 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:15:00.013 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:15:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:15:00.176 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:15:00.533 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-25T01:15:00.693 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-25T01:15:00.694 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-25T01:15:00.908 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:15:00.957 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:15:00.990 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:15:01.089 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:15:01.148 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:15:01.236 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:15:01.267 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:15:01.472 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:15:01.498 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:15:01.576 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:15:02.091 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:15:02.091 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-04-25T01:15:02.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:02.234 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:15:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:15:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:15:02.280 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:02.280 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:15:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:02.357 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:15:02.579 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-25T01:15:02.579 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-25T01:15:03.590 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:15:03.678 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:15:03.710 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:15:03.739 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:15:03.817 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:04.052 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:15:04.084 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:15:04.093 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:15:04.108 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:15:04.108 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:04.108 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:15:04.108 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:15:04.108 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:15:04.108 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:04.184 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:15:04.215 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:15:04.251 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:15:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:15:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:15:04.600 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:04.600 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:15:04.600 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:15:04.600 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:15:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:04.707 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:15:04.722 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:15:04.754 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:15:04.756 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:15:04.809 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:15:04.820 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:15:04.840 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:15:05.018 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:15:05.069 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:15:05.142 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:15:05.232 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:05.377 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:05.419 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:15:05.451 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:15:05.478 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:15:05.929 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:15:05.960 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:15:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:15:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:15:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:15:07.087 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:15:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:15:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:15:07.344 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:15:07.883 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:15:07.939 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:15:08.041 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:15:08.041 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:15:08.041 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:15:08.041 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:08.074 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:15:08.239 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:15:08.268 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:15:08.376 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:15:08.539 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:15:08.562 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:15:08.587 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:15:08.587 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:08.587 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:15:08.587 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:15:08.587 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:15:08.587 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:08.646 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:15:08.658 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:15:08.685 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-25T01:15:08.759 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:15:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:15:08.813 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:15:08.829 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:15:08.883 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:15:08.925 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:15:08.950 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:15:08.994 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:15:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:15:09.451 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:09.603 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:09.688 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:15:09.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:15:09.790 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:15:09.814 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:15:09.814 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:09.814 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:15:09.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:15:09.814 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:15:09.815 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:09.858 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:15:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:15:09.881 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:15:09.881 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:09.881 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:15:09.882 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:09.913 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:15:09.940 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:09.942 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:15:09.965 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:15:09.965 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:09.966 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:15:09.966 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:15:09.966 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:15:09.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:10.046 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:10.075 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:15:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:15:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:15:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:15:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:15:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:15:10.309 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:10.309 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:15:10.309 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:15:10.309 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:15:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:10.386 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:15:10.418 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:15:10.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:15:10.643 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:15:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:15:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:15:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:15:10.842 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:15:10.886 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:15:10.918 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:15:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:15:11.017 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:15:11.078 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:15:11.108 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:15:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:15:11.221 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:15:11.254 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:15:11.298 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:15:11.326 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:15:11.399 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:15:11.441 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:15:11.527 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:15:11.671 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:15:11.737 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:15:12.287 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:15:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:15:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:15:12.462 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:15:12.500 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:15:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:15:12.720 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:15:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:15:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:15:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:15:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:15:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:15:13.037 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:15:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:15:13.117 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:15:13.158 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:15:13.303 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:15:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:15:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:15:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:15:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:15:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:15:13.593 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:15:13.618 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:15:13.721 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:15:13.850 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:15:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:15:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:15:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:15:13.994 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:15:14.034 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:15:14.064 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:15:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:15:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:15:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:15:14.209 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:15:14.298 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:15:14.347 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:15:14.386 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:15:14.434 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:15:14.544 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:15:14.597 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:15:15.089 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:15:15.130 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:15:15.172 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:15:15.203 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:15:15.230 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:15:15.358 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:15:15.402 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:15:15.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:15:15.436 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:15:15.476 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:15:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:15:15.538 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:15:15.595 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:15:15.685 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:15:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:15:15.907 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:15:16.097 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:15:16.097 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:15:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:16.710 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:15:16.830 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:15:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:15:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:15:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:15:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:15:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:15:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:15:18.200 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:15:18.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:15:18.262 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:15:18.588 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:15:18.632 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:15:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:15:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:15:18.791 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:15:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:15:19.394 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:15:19.465 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:15:19.495 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:15:19.632 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:15:19.632 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-04-25T01:15:19.632 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-04-25T01:15:19.632 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-04-25T01:15:19.633 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-04-25T01:15:19.633 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-04-25T01:15:19.633 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-04-25T01:15:19.633 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-04-25T01:15:19.633 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:19.796 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:15:19.812 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:15:19.812 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:19.812 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:15:19.812 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:21.291 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:15:21.317 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:15:21.317 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:21.317 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:15:21.317 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:15:21.318 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:15:21.318 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:21.542 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:15:21.565 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:15:21.565 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:21.565 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:15:21.565 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:15:21.565 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:15:21.565 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:15:21.846 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:15:22.009 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:15:22.031 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:15:22.031 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:22.031 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:15:22.032 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:15:22.032 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:15:22.032 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:22.139 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:15:22.196 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:15:22.250 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:15:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:15:22.439 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:15:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:15:22.493 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:15:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:15:22.539 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:15:22.539 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:15:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:22.571 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:15:22.574 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:15:22.660 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:22.801 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:22.835 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:15:22.867 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:15:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:15:23.285 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:15:23.309 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:15:23.341 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:15:23.403 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:15:23.433 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:23.657 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:15:23.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:15:23.659 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:15:23.670 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:15:23.671 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:15:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:15:23.673 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:15:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:15:23.675 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:15:23.675 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:15:23.678 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:15:23.678 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:15:23.678 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:15:23.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:15:23.681 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:15:23.682 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:23.683 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:15:24.472 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:15:24.501 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:15:25.474 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:15:25.501 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:15:25.605 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:15:25.765 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:15:25.795 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:15:25.818 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:15:25.819 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:25.819 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:15:25.819 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:15:25.819 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:15:25.819 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:25.871 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:15:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:15:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:15:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:15:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:15:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:15:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:15:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:15:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:15:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:15:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:15:25.890 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:15:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:15:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:15:25.891 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-25T01:15:25.980 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:15:26.027 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:15:26.264 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:26.927 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:15:26.957 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:15:26.980 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:15:26.980 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:26.980 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:15:26.980 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:15:26.980 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:15:26.980 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:27.024 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:15:27.046 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:15:27.046 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:27.046 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:15:27.046 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:27.091 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:15:27.114 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:15:27.114 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:27.114 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:15:27.114 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:15:27.114 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:15:27.114 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:27.425 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:15:27.448 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:15:27.448 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:27.448 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:15:27.448 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:15:27.448 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:15:27.448 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:31.987 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:15:32.046 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:15:32.086 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:15:32.118 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:15:32.146 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:15:32.326 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:15:32.367 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:15:32.397 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:15:33.031 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:33.070 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:33.090 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:33.938 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:15:34.024 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:15:34.066 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:15:34.147 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:15:34.179 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:15:34.179 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:34.255 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:34.302 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:34.303 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:34.362 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:34.428 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-25T01:15:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:15:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:15:34.831 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:34.831 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:15:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:36.350 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:15:36.378 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:15:36.378 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:36.378 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:15:36.378 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:15:36.378 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:15:36.378 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:36.609 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:15:36.633 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:15:36.633 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:36.633 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:15:36.633 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:15:36.633 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:15:36.634 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:37.110 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:15:37.135 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:15:37.135 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:15:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:15:37.136 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:15:37.136 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:15:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:15:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:15:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:15:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:15:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:15:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:15:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:15:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:15:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:15:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:15:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:15:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:15:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:15:39.712 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:15:39.745 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:15:40.420 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:15:40.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:40.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:40.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:15:40.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:15:40.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:15:40.421 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:40.426 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:15:40.426 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:15:40.426 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:15:40.426 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:15:40.426 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:15:40.427 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:15:40.428 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:15:40.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:15:40.430 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:15:40.431 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:15:40.432 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:15:40.433 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:40.434 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:15:40.435 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:15:40.436 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:15:40.437 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:40.438 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:15:40.765 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:15:40.796 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:15:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:15:41.066 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:15:41.098 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:15:41.123 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:15:41.123 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:41.123 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:15:41.123 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:15:41.123 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:15:41.123 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:41.174 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:15:41.185 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:15:41.287 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:15:41.340 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:15:42.285 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:15:42.318 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:15:42.342 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:15:42.342 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:42.343 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:15:42.343 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:15:42.343 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:15:42.343 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:42.386 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:15:42.409 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:15:42.409 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:42.409 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:15:42.409 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:42.458 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:42.458 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.459 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.460 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:15:42.461 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:15:42.462 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:15:42.463 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:15:42.464 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:15:42.465 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:15:42.466 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:15:42.467 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.468 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:15:42.469 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:15:42.470 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:15:42.471 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:15:42.471 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:15:42.471 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:15:42.471 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-25T01:15:42.485 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:15:42.485 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:42.485 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:15:42.486 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:15:42.486 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:15:42.486 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:42.804 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:15:42.828 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:15:42.828 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:15:42.828 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:15:42.828 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:15:42.828 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:15:42.828 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:42.895 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:47.583 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:15:47.625 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:15:47.666 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:15:47.706 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:15:47.733 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:15:47.927 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:15:47.980 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:15:48.010 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:15:48.645 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:48.683 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:48.704 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:49.614 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:15:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:15:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:15:49.839 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:15:49.869 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:15:49.869 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:49.985 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:49.985 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:50.078 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:50.155 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:15:55.937 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:15:55.938 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:15:55.939 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:15:55.940 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:15:55.941 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:15:55.942 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:15:55.943 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:15:55.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:15:55.945 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:15:55.946 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:15:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:55.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-25T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-25T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-25T01:15:58.501 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:58.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:15:58.652 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:15:58.679 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:58.680 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:58.680 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:15:58.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:15:58.902 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:15:58.924 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:58.924 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:58.924 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:15:58.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:15:59.070 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:15:59.095 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:59.095 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:59.095 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:15:59.097 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:15:59.097 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:15:59.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:15:59.130 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:15:59.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:15:59.157 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:15:59.157 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:15:59.187 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:15:59.187 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:15:59.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:15:59.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:15:59.284 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:15:59.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:15:59.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:15:59.377 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:15:59.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:15:59.406 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:15:59.474 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:15:59.474 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:15:59.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:15:59.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:15:59.572 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:15:59.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:15:59.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:15:59.667 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:15:59.667 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:15:59.691 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:15:59.755 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:15:59.755 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:15:59.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:15:59.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:15:59.846 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:15:59.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:15:59.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:15:59.936 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:15:59.936 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:15:59.961 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:16:00.025 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T01:16:00.141 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-25T01:16:00.142 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:16:00.142 INFO:tasks.cephadm:Cluster fsid is 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:00.142 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T01:16:00.142 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.38', 'mon.c': '172.21.15.156'} 2024-04-25T01:16:00.143 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-04-25T01:16:00.143 INFO:tasks.cephadm:First mgr is x 2024-04-25T01:16:00.143 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T01:16:00.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-04-25T01:16:00.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-25T01:16:00.193 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2024-04-25T01:16:00.217 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T01:16:00.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:16:00.359 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T01:16:00.575 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:16:00.577 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-25T01:16:00.577 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:00.958 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:16 /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:00.958 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:16 /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.349 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.726 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:16 /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.727 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.747 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.768 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:16:01.810 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-25T01:16:01.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:16:01.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:16:01.821 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:16:03.286 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:16:03.317 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:16:03.810 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:16:31.489 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:16:31.490 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:16:31.490 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:16:31.490 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-04-25T01:16:31.490 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:16:31.490 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:16:31.490 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-25T01:16:38.101 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-04-25T01:16:42.885 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T01:16:42.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-04-25T01:16:42.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-25T01:16:42.974 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2024-04-25T01:16:43.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-04-25T01:16:43.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-25T01:16:43.058 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2024-04-25T01:16:43.082 INFO:tasks.cephadm:Writing seed config... 2024-04-25T01:16:43.083 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T01:16:43.083 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T01:16:43.083 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T01:16:43.083 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T01:16:43.083 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T01:16:43.083 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T01:16:43.084 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T01:16:43.085 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T01:16:43.086 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T01:16:43.087 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:16:43.087 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T01:16:43.106 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 605c42f8-02a1-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T01:16:43.106 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T01:16:43.149 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T01:16:43.193 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T01:16:43.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:16:43.395 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:16:43.395 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '605c42f8-02a1-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-04-25T01:16:43.396 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T01:16:43.396 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-04-25T01:16:43.422 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:16:43.422 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-04-25T01:16:43.423 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-04-25T01:16:43.430 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:16:43.430 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:16:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:16:43.436 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-25T01:16:43.443 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-25T01:16:43.449 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-25T01:16:43.449 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:16:43.449 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-04-25T01:16:43.473 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:16:43.473 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T01:16:43.473 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-04-25T01:16:43.473 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-04-25T01:16:43.479 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:16:43.479 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:16:43.485 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:16:43.486 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-25T01:16:43.492 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-25T01:16:43.498 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 140497371542432 on /run/cephadm/605c42f8-02a1-11ef-bc93-c7b262605968.lock 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 140497371542432 acquired on /run/cephadm/605c42f8-02a1-11ef-bc93-c7b262605968.lock 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-04-25T01:16:43.499 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-04-25T01:16:43.500 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-25T01:16:43.503 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-04-25T01:16:43.503 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-04-25T01:16:43.505 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T01:16:43.505 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T01:16:43.505 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T01:16:43.508 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T01:16:43.508 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T01:16:43.508 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:16:43.508 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T01:16:43.508 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-25T01:16:43.508 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:16:43.509 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-25T01:16:43.509 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-25T01:16:43.509 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T01:16:43.509 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T01:16:43.510 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:16:43.990 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T01:16:44.912 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:16:44.912 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:16:44.912 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T01:16:45.768 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-04-25T01:16:45.768 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-04-25T01:16:46.699 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQD+rilmCqKtBxAAzzH71QuN/+WiqI2mKEEXgg== 2024-04-25T01:16:47.629 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQD/rilm42CBAxAA3ByEyo+6UnZQKPvsfJ2Kpg== 2024-04-25T01:16:48.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQD/rilm+M+GOxAAlQlyFY+b7lGnHWVlgetKaw== 2024-04-25T01:16:48.719 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-04-25T01:16:55.123 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:16:55.123 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T01:16:55.123 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:55.123 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:16:55.123 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:16:55.123 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-04-25T01:16:55.124 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:16:55.124 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:16:55.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:16:55.124 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-04-25T01:17:01.334 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-04-25T01:17:01.566 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T01:17:01.718 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T01:17:01.902 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-605c42f8-02a1-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-605c42f8-02a1-11ef-bc93-c7b262605968.target. 2024-04-25T01:17:01.902 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-605c42f8-02a1-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-605c42f8-02a1-11ef-bc93-c7b262605968.target. 2024-04-25T01:17:02.350 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:17:02.350 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T01:17:02.520 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-605c42f8-02a1-11ef-bc93-c7b262605968.target.wants/ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-605c42f8-02a1-11ef-bc93-c7b262605968@.service. 2024-04-25T01:17:02.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:02 smithi016 podman[28765]: 2024-04-25 01:17:02.678898259 +0000 UTC m=+0.018012895 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:17:03.000 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:17:03.000 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-25T01:17:03.010 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:17:03.010 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-25T01:17:03.010 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-25T01:17:03.010 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:17:03.010 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-04-25T01:17:03.011 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-04-25T01:17:03.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:02 smithi016 podman[28765]: 2024-04-25 01:17:02.927509007 +0000 UTC m=+0.266623634 container init 2539920d1d1ea8dbd1b8ec2308329273484e5212b715e21f6570f03bf0f5148c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:17:03.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:02 smithi016 podman[28765]: 2024-04-25 01:17:02.930736205 +0000 UTC m=+0.269850835 container start 2539920d1d1ea8dbd1b8ec2308329273484e5212b715e21f6570f03bf0f5148c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T01:17:03.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:02 smithi016 ceph-mon[28784]: mkfs 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:03.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:02 smithi016 ceph-mon[28784]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:17:03.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 bash[28765]: 2539920d1d1ea8dbd1b8ec2308329273484e5212b715e21f6570f03bf0f5148c 2024-04-25T01:17:03.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 systemd[1]: Started Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:17:04.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:17:04.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: monmap epoch 1 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: last_changed 2024-04-25T01:16:49.048092+0000 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: min_mon_release 19 (squid) 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: election_strategy: 1 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: fsmap 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: mgrmap e1: no daemons active 2024-04-25T01:17:04.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:03 smithi016 ceph-mon[28784]: from='client.? 172.21.15.16:0/487657904' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T01:17:04.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T01:17:04.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:04.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T01:17:04.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.554653s) 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:04.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.447+0000 7f3a0e0c9700 1 Processor -- start 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a0e0c9700 1 -- start start 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a0e0c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0807c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a0e0c9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0807c870 con 0x7f3a0807de40 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a077fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0807c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a077fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0807c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43386/0 (socket says 172.21.15.16:43386) 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a077fe700 1 -- 172.21.15.16:0/664526473 learned_addr learned my addr 172.21.15.16:0/664526473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.448+0000 7f3a077fe700 1 -- 172.21.15.16:0/664526473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0807c9f0 con 0x7f3a0807de40 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a077fe700 1 --2- 172.21.15.16:0/664526473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0807c2a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f39f00098a0 tx=0x7f39f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc8b337dbba2666 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a067fc700 1 -- 172.21.15.16:0/664526473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39f0004160 con 0x7f3a0807de40 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a067fc700 1 -- 172.21.15.16:0/664526473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f0004300 con 0x7f3a0807de40 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a067fc700 1 -- 172.21.15.16:0/664526473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39f00036a0 con 0x7f3a0807de40 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/664526473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 msgr2=0x7f3a0807c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 --2- 172.21.15.16:0/664526473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0807c2a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f39f00098a0 tx=0x7f39f0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/664526473 shutdown_connections 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 --2- 172.21.15.16:0/664526473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0807c2a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/664526473 >> 172.21.15.16:0/664526473 conn(0x7f3a08103180 msgr2=0x7f3a081055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/664526473 shutdown_connections 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.449+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/664526473 wait complete. 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a0e0c9700 1 Processor -- start 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a0e0c9700 1 -- start start 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a0e0c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0819d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a0e0c9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0807cf20 con 0x7f3a0807de40 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a077fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0819d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a077fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0819d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43388/0 (socket says 172.21.15.16:43388) 2024-04-25T01:17:04.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.450+0000 7f3a077fe700 1 -- 172.21.15.16:0/487657904 learned_addr learned my addr 172.21.15.16:0/487657904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a077fe700 1 -- 172.21.15.16:0/487657904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0819d9f0 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a077fe700 1 --2- 172.21.15.16:0/487657904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0819d4b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39f0000c00 tx=0x7f39f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39f0003db0 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f0011630 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0819dc80 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39f0011950 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a081a0970 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f39f0011af0 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.451+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f39f001c450 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.452+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a08063890 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.455+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f39f0032020 con 0x7f3a0807de40 2024-04-25T01:17:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.527+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f3a0802dcb0 con 0x7f3a0807de40 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.527+0000 7f3a04ff9700 1 -- 172.21.15.16:0/487657904 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f39f0004260 con 0x7f3a0807de40 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 msgr2=0x7f3a0819d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 --2- 172.21.15.16:0/487657904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0819d4b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39f0000c00 tx=0x7f39f0004750 comp rx=0 tx=0).stop 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 shutdown_connections 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 --2- 172.21.15.16:0/487657904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a0807de40 0x7f3a0819d4b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 >> 172.21.15.16:0/487657904 conn(0x7f3a08103180 msgr2=0x7f3a08105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 shutdown_connections 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:03.528+0000 7f3a0e0c9700 1 -- 172.21.15.16:0/487657904 wait complete. 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-04-25T01:17:04.065 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T01:17:05.060 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:17:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e259da700 1 Processor -- start 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e259da700 1 -- start start 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e259da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e259da700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2007c870 con 0x7f2e2007de40 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43400/0 (socket says 172.21.15.16:43400) 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e1effd700 1 -- 172.21.15.16:0/1926096195 learned_addr learned my addr 172.21.15.16:0/1926096195 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.446+0000 7f2e1effd700 1 -- 172.21.15.16:0/1926096195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2007c9f0 con 0x7f2e2007de40 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e1effd700 1 --2- 172.21.15.16:0/1926096195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2007c2a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2e100098a0 tx=0x7f2e10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77d04a883885b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e1dffb700 1 -- 172.21.15.16:0/1926096195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e10004160 con 0x7f2e2007de40 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e1dffb700 1 -- 172.21.15.16:0/1926096195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10004300 con 0x7f2e2007de40 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e1dffb700 1 -- 172.21.15.16:0/1926096195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e100036a0 con 0x7f2e2007de40 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 -- 172.21.15.16:0/1926096195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 msgr2=0x7f2e2007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 --2- 172.21.15.16:0/1926096195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2007c2a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2e100098a0 tx=0x7f2e10009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 -- 172.21.15.16:0/1926096195 shutdown_connections 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 --2- 172.21.15.16:0/1926096195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2007c2a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 -- 172.21.15.16:0/1926096195 >> 172.21.15.16:0/1926096195 conn(0x7f2e20103180 msgr2=0x7f2e201055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 -- 172.21.15.16:0/1926096195 shutdown_connections 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.447+0000 7f2e259da700 1 -- 172.21.15.16:0/1926096195 wait complete. 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.448+0000 7f2e259da700 1 Processor -- start 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.448+0000 7f2e259da700 1 -- start start 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.448+0000 7f2e259da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.448+0000 7f2e259da700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2007cf20 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43410/0 (socket says 172.21.15.16:43410) 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e1effd700 1 -- 172.21.15.16:0/152192430 learned_addr learned my addr 172.21.15.16:0/152192430 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e1effd700 1 -- 172.21.15.16:0/152192430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2019d9f0 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e1effd700 1 --2- 172.21.15.16:0/152192430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2019d4b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c00 tx=0x7f2e10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e10003db0 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10011630 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2019dc80 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e201a0970 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e10011950 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2e10011b30 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.449+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2e1001c450 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.450+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e0c005360 con 0x7f2e2007de40 2024-04-25T01:17:05.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.452+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2e10032020 con 0x7f2e2007de40 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.523+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2e0c002430 con 0x7f2e2007de40 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.526+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f2e10004260 con 0x7f2e2007de40 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.527+0000 7f2e249d8700 1 -- 172.21.15.16:0/152192430 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2e1001c6a0 con 0x7f2e2007de40 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.527+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 msgr2=0x7f2e2019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.527+0000 7f2e259da700 1 --2- 172.21.15.16:0/152192430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2019d4b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c00 tx=0x7f2e10004750 comp rx=0 tx=0).stop 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.527+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 shutdown_connections 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.527+0000 7f2e259da700 1 --2- 172.21.15.16:0/152192430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2007de40 0x7f2e2019d4b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.527+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 >> 172.21.15.16:0/152192430 conn(0x7f2e20103180 msgr2=0x7f2e20105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.528+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 shutdown_connections 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:04.528+0000 7f2e259da700 1 -- 172.21.15.16:0/152192430 wait complete. 2024-04-25T01:17:05.065 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-04-25T01:17:05.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:04 smithi016 ceph-mon[28784]: from='client.? 172.21.15.16:0/152192430' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:17:05.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:04 smithi016 ceph-mon[28784]: from='client.? 172.21.15.16:0/152192430' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T01:17:06.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:05 smithi016 ceph-mon[28784]: from='client.? 172.21.15.16:0/4044577068' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:17:06.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa50b008700 1 Processor -- start 2024-04-25T01:17:06.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa50b008700 1 -- start start 2024-04-25T01:17:06.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa50b008700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa504107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa50b008700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa504108590 con 0x7fa504107ba0 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa508da4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa504107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa508da4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa504107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43418/0 (socket says 172.21.15.16:43418) 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa508da4700 1 -- 172.21.15.16:0/3351422621 learned_addr learned my addr 172.21.15.16:0/3351422621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.481+0000 7fa508da4700 1 -- 172.21.15.16:0/3351422621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa504108da0 con 0x7fa504107ba0 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa508da4700 1 --2- 172.21.15.16:0/3351422621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa504107fc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80098a0 tx=0x7fa4f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c5cea877f04d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa5037fe700 1 -- 172.21.15.16:0/3351422621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4f8004160 con 0x7fa504107ba0 2024-04-25T01:17:06.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa5037fe700 1 -- 172.21.15.16:0/3351422621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa4f8004300 con 0x7fa504107ba0 2024-04-25T01:17:06.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa5037fe700 1 -- 172.21.15.16:0/3351422621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4f8011a80 con 0x7fa504107ba0 2024-04-25T01:17:06.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 -- 172.21.15.16:0/3351422621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 msgr2=0x7fa504107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 --2- 172.21.15.16:0/3351422621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa504107fc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80098a0 tx=0x7fa4f8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 -- 172.21.15.16:0/3351422621 shutdown_connections 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 --2- 172.21.15.16:0/3351422621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa504107fc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 -- 172.21.15.16:0/3351422621 >> 172.21.15.16:0/3351422621 conn(0x7fa504103180 msgr2=0x7fa5041055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 -- 172.21.15.16:0/3351422621 shutdown_connections 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.482+0000 7fa50b008700 1 -- 172.21.15.16:0/3351422621 wait complete. 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa50b008700 1 Processor -- start 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa50b008700 1 -- start start 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa50b008700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa50407db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa50b008700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa50410c150 con 0x7fa504107ba0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa508da4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa50407db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa508da4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa50407db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43426/0 (socket says 172.21.15.16:43426) 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.483+0000 7fa508da4700 1 -- 172.21.15.16:0/4044577068 learned_addr learned my addr 172.21.15.16:0/4044577068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa508da4700 1 -- 172.21.15.16:0/4044577068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50407e050 con 0x7fa504107ba0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa508da4700 1 --2- 172.21.15.16:0/4044577068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa50407db10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8006af0 tx=0x7fa4f8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4f80039e0 con 0x7fa504107ba0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa4f8003b80 con 0x7fa504107ba0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50407c240 con 0x7fa504107ba0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4f80196a0 con 0x7fa504107ba0 2024-04-25T01:17:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50407c780 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4f802d440 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.484+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa4f80247c0 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.485+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa504063890 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.487+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa4f8043dd0 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.558+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fa50407ca30 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.558+0000 7fa501ffb700 1 -- 172.21.15.16:0/4044577068 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fa4f8032020 con 0x7fa504107ba0 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 msgr2=0x7fa50407db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 --2- 172.21.15.16:0/4044577068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa50407db10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8006af0 tx=0x7fa4f8011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 shutdown_connections 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 --2- 172.21.15.16:0/4044577068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa504107ba0 0x7fa50407db10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 >> 172.21.15.16:0/4044577068 conn(0x7fa504103180 msgr2=0x7fa504103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:06.400 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 shutdown_connections 2024-04-25T01:17:06.401 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:05.559+0000 7fa50b008700 1 -- 172.21.15.16:0/4044577068 wait complete. 2024-04-25T01:17:06.401 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-04-25T01:17:06.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:06 smithi016 systemd[1]: Stopping Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:17:07.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:07 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a[28780]: 2024-04-25T01:17:07.566+0000 7f7959849700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:17:07.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:07 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a[28780]: 2024-04-25T01:17:07.566+0000 7f7959849700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T01:17:07.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:07 smithi016 podman[29029]: 2024-04-25 01:17:07.700177747 +0000 UTC m=+1.111026124 container died 2539920d1d1ea8dbd1b8ec2308329273484e5212b715e21f6570f03bf0f5148c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T01:17:08.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 podman[29029]: 2024-04-25 01:17:08.289426869 +0000 UTC m=+1.700275249 container cleanup 2539920d1d1ea8dbd1b8ec2308329273484e5212b715e21f6570f03bf0f5148c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:17:08.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 bash[29029]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:17:08.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 podman[29043]: 2024-04-25 01:17:08.53189051 +0000 UTC m=+0.829788588 container remove 2539920d1d1ea8dbd1b8ec2308329273484e5212b715e21f6570f03bf0f5148c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T01:17:08.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:17:08.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 systemd[1]: Stopped Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:17:08.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 systemd[1]: Starting Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:17:09.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 podman[29134]: 2024-04-25 01:17:08.857726323 +0000 UTC m=+0.101543524 container create 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T01:17:09.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:08 smithi016 podman[29134]: 2024-04-25 01:17:08.774437517 +0000 UTC m=+0.018254731 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:17:09.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 podman[29134]: 2024-04-25 01:17:09.076788534 +0000 UTC m=+0.320605744 container init 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:17:09.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 podman[29134]: 2024-04-25 01:17:09.080169613 +0000 UTC m=+0.323986814 container start 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:17:09.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:17:09.093 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:17:09.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: pidfile_write: ignore empty --pid-file 2024-04-25T01:17:09.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: load: jerasure load: lrc 2024-04-25T01:17:09.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:17:09.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Git sha 0 2024-04-25T01:17:09.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:17:09.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: DB SUMMARY 2024-04-25T01:17:09.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: DB Session ID: N7J77VGGPX0IKOWGMQBA 2024-04-25T01:17:09.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:17:09.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:17:09.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T01:17:09.095 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T01:17:09.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104898 ; 2024-04-25T01:17:09.096 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:17:09.165 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T01:17:09.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:17:09.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:17:09.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:17:09.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:17:09.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:17:09.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.env: 0x55e3b0be7c60 2024-04-25T01:17:09.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:17:09.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.info_log: 0x55e3b2f03aa0 2024-04-25T01:17:09.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:17:09.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.statistics: (nil) 2024-04-25T01:17:09.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.use_fsync: 0 2024-04-25T01:17:09.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:17:09.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:17:09.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:17:09.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:17:09.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:17:09.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:17:09.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:17:09.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:17:09.368 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:17:09.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:17:09.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:17:09.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.db_log_dir: 2024-04-25T01:17:09.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.wal_dir: 2024-04-25T01:17:09.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:17:09.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:17:09.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:17:09.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:17:09.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:17:09.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:17:09.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:17:09.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:17:09.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.write_buffer_manager: 0x55e3b2ed61e0 2024-04-25T01:17:09.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:17:09.372 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:17:09.373 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.unordered_write: 0 2024-04-25T01:17:09.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:17:09.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:17:09.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:17:09.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:17:09.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.row_cache: None 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.wal_filter: None 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.wal_compression: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:17:09.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:17:09.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_open_files: -1 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:17:09.377 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:17:09.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:17:09.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:17:09.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Compression algorithms supported: 2024-04-25T01:17:09.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:17:09.378 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kZSTD supported: 0 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T01:17:09.379 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.merge_operator: 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_filter: None 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:17:09.380 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3b2f03ba0) 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-04-25T01:17:09.381 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55e3b2f07090 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-04-25T01:17:09.382 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-04-25T01:17:09.383 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-04-25T01:17:09.384 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-04-25T01:17:09.384 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-04-25T01:17:09.384 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-04-25T01:17:09.384 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-04-25T01:17:09.385 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-04-25T01:17:09.385 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-04-25T01:17:09.385 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-04-25T01:17:09.385 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:17:09.385 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-04-25T01:17:09.385 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression: NoCompression 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:17:09.386 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.num_levels: 7 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:17:09.387 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:17:09.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:17:09.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:17:09.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:17:09.391 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:17:09.392 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.ttl: 2592000 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:17:09.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 121bc70d-7494-4cb3-b6f2-82e7af66a97e 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007829109515, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T01:17:09.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007829110632, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100020, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97931, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14193, "raw_average_key_size": 49, "raw_value_size": 90210, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714007829, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "121bc70d-7494-4cb3-b6f2-82e7af66a97e", "db_session_id": "N7J77VGGPX0IKOWGMQBA", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007829110716, "job": 1, "event": "recovery_finished"} 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e3b2ff8000 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: DB pointer 0x55e3b2fe2000 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:17:09.395 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 99.51 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 99.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:17:09.396 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 29.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 29.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x55e3b2f07090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-25T01:17:09.397 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???) e1 preinit fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).mds e1 new map 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).mds e1 print_map 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:17:09.398 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: monmap epoch 1 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:09.399 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: last_changed 2024-04-25T01:16:49.048092+0000 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: min_mon_release 19 (squid) 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: election_strategy: 1 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: fsmap 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 ceph-mon[29153]: mgrmap e1: no daemons active 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 bash[29134]: 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb 2024-04-25T01:17:09.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:09 smithi016 systemd[1]: Started Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:17:10.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.586+0000 7fea8ddcb700 1 Processor -- start 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.586+0000 7fea8ddcb700 1 -- start start 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.586+0000 7fea8ddcb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea8810aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.586+0000 7fea8ddcb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea8807bdb0 con 0x7fea881086b0 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea8810aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea8810aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43436/0 (socket says 172.21.15.16:43436) 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea877fe700 1 -- 172.21.15.16:0/3987739580 learned_addr learned my addr 172.21.15.16:0/3987739580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea877fe700 1 -- 172.21.15.16:0/3987739580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8810afe0 con 0x7fea881086b0 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea877fe700 1 --2- 172.21.15.16:0/3987739580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea8810aaa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fea780098a0 tx=0x7fea78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cca3186e9f66de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea867fc700 1 -- 172.21.15.16:0/3987739580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78004160 con 0x7fea881086b0 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea867fc700 1 -- 172.21.15.16:0/3987739580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fea78004300 con 0x7fea881086b0 2024-04-25T01:17:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.587+0000 7fea867fc700 1 -- 172.21.15.16:0/3987739580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78011a80 con 0x7fea881086b0 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 -- 172.21.15.16:0/3987739580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 msgr2=0x7fea8810aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 --2- 172.21.15.16:0/3987739580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea8810aaa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fea780098a0 tx=0x7fea78009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 -- 172.21.15.16:0/3987739580 shutdown_connections 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 --2- 172.21.15.16:0/3987739580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea8810aaa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 -- 172.21.15.16:0/3987739580 >> 172.21.15.16:0/3987739580 conn(0x7fea88102060 msgr2=0x7fea88104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 -- 172.21.15.16:0/3987739580 shutdown_connections 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.588+0000 7fea8ddcb700 1 -- 172.21.15.16:0/3987739580 wait complete. 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea8ddcb700 1 Processor -- start 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea8ddcb700 1 -- start start 2024-04-25T01:17:10.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea8ddcb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea88107f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea8ddcb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810bf40 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea88107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea88107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43448/0 (socket says 172.21.15.16:43448) 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea877fe700 1 -- 172.21.15.16:0/4051333619 learned_addr learned my addr 172.21.15.16:0/4051333619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea877fe700 1 -- 172.21.15.16:0/4051333619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea88108470 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.589+0000 7fea877fe700 1 --2- 172.21.15.16:0/4051333619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea88107f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea78011a50 tx=0x7fea78011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78003a80 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fea78003c20 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea88106660 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea7801a650 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea88106ba0 con 0x7fea881086b0 2024-04-25T01:17:10.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea7801ad00 con 0x7fea881086b0 2024-04-25T01:17:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.590+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea78019700 con 0x7fea881086b0 2024-04-25T01:17:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.591+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea88063890 con 0x7fea881086b0 2024-04-25T01:17:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.593+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fea78024ad0 con 0x7fea881086b0 2024-04-25T01:17:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.663+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fea88106ee0 con 0x7fea881086b0 2024-04-25T01:17:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.665+0000 7fea84ff9700 1 -- 172.21.15.16:0/4051333619 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fea78024440 con 0x7fea881086b0 2024-04-25T01:17:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 msgr2=0x7fea88107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 --2- 172.21.15.16:0/4051333619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea88107f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea78011a50 tx=0x7fea78011b30 comp rx=0 tx=0).stop 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 shutdown_connections 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 --2- 172.21.15.16:0/4051333619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea881086b0 0x7fea88107f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 >> 172.21.15.16:0/4051333619 conn(0x7fea88102060 msgr2=0x7fea88102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 shutdown_connections 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:09.666+0000 7fea8ddcb700 1 -- 172.21.15.16:0/4051333619 wait complete. 2024-04-25T01:17:10.204 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T01:17:10.224 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:17:10.224 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-04-25T01:17:10.225 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T01:17:10.225 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T01:17:10.479 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:17:10.479 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T01:17:10.644 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-605c42f8-02a1-11ef-bc93-c7b262605968.target.wants/ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-605c42f8-02a1-11ef-bc93-c7b262605968@.service. 2024-04-25T01:17:10.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:10 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4051333619' entity='client.admin' 2024-04-25T01:17:10.865 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:10 smithi016 systemd[1]: Starting Ceph mgr.x for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:17:11.213 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:10 smithi016 podman[29371]: 2024-04-25 01:17:10.807272548 +0000 UTC m=+0.018167375 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:17:11.213 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:10 smithi016 podman[29371]: 2024-04-25 01:17:10.918765054 +0000 UTC m=+0.129659865 container create 4c4a282d44c92ab066678b2930b317866cb10b19541c154827f4c68f6e9eee8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T01:17:11.213 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 podman[29371]: 2024-04-25 01:17:11.146314384 +0000 UTC m=+0.357209194 container init 4c4a282d44c92ab066678b2930b317866cb10b19541c154827f4c68f6e9eee8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T01:17:11.213 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 podman[29371]: 2024-04-25 01:17:11.148881063 +0000 UTC m=+0.359775876 container start 4c4a282d44c92ab066678b2930b317866cb10b19541c154827f4c68f6e9eee8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T01:17:11.213 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 bash[29371]: 4c4a282d44c92ab066678b2930b317866cb10b19541c154827f4c68f6e9eee8a 2024-04-25T01:17:11.227 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:17:11.227 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-25T01:17:11.235 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:17:11.235 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-25T01:17:11.235 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-25T01:17:11.235 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:17:11.244 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:17:11.245 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-25T01:17:11.253 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:17:11.253 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-25T01:17:11.253 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-25T01:17:11.253 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T01:17:11.253 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-04-25T01:17:11.253 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-04-25T01:17:11.490 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 systemd[1]: Started Ceph mgr.x for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:17:11.490 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:11.315+0000 7ff38ddf3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:17:11.782 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:11.519+0000 7ff38ddf3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:11.783 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:11.625+0000 7ff38ddf3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:17:12.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:11 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3997280382' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:17:12.086 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:11 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:11.957+0000 7ff38ddf3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:17:12.086 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:12 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:12.046+0000 7ff38ddf3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:17:12.376 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:12.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:17:12.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:17:12.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:17:02.975509+0000", 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.718+0000 7ff33136d700 1 Processor -- start 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.718+0000 7ff33136d700 1 -- start start 2024-04-25T01:17:12.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.718+0000 7ff33136d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.718+0000 7ff33136d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c07c890 con 0x7ff32c07de60 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff32affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff32affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45668/0 (socket says 172.21.15.16:45668) 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff32affd700 1 -- 172.21.15.16:0/3714678871 learned_addr learned my addr 172.21.15.16:0/3714678871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff32affd700 1 -- 172.21.15.16:0/3714678871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c07ca10 con 0x7ff32c07de60 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff32affd700 1 --2- 172.21.15.16:0/3714678871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0098a0 tx=0x7ff31c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42023f0e3b12a8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff329ffb700 1 -- 172.21.15.16:0/3714678871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff31c004160 con 0x7ff32c07de60 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff329ffb700 1 -- 172.21.15.16:0/3714678871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff31c004300 con 0x7ff32c07de60 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.719+0000 7ff329ffb700 1 -- 172.21.15.16:0/3714678871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff31c011a80 con 0x7ff32c07de60 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 -- 172.21.15.16:0/3714678871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 msgr2=0x7ff32c07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:12.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 --2- 172.21.15.16:0/3714678871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0098a0 tx=0x7ff31c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 -- 172.21.15.16:0/3714678871 shutdown_connections 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 --2- 172.21.15.16:0/3714678871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c07c2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 -- 172.21.15.16:0/3714678871 >> 172.21.15.16:0/3714678871 conn(0x7ff32c103110 msgr2=0x7ff32c105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 -- 172.21.15.16:0/3714678871 shutdown_connections 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 -- 172.21.15.16:0/3714678871 wait complete. 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.720+0000 7ff33136d700 1 Processor -- start 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff33136d700 1 -- start start 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff33136d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c19dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff33136d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c10e0c0 con 0x7ff32c07de60 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff32affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c19dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff32affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c19dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45676/0 (socket says 172.21.15.16:45676) 2024-04-25T01:17:12.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff32affd700 1 -- 172.21.15.16:0/3997280382 learned_addr learned my addr 172.21.15.16:0/3997280382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff32affd700 1 -- 172.21.15.16:0/3997280382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c19e1a0 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff32affd700 1 --2- 172.21.15.16:0/3997280382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c19dc60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff31c006af0 tx=0x7ff31c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.721+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff31c0039e0 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff31c003b80 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff31c0196a0 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c19e430 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c1a1120 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff31c02d440 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff31c024770 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.722+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff32c063890 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.725+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff31c043dd0 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.798+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff32c1a12f0 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.799+0000 7ff323fff700 1 -- 172.21.15.16:0/3997280382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff31c032020 con 0x7ff32c07de60 2024-04-25T01:17:12.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 msgr2=0x7ff32c19dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 --2- 172.21.15.16:0/3997280382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c19dc60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff31c006af0 tx=0x7ff31c011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 shutdown_connections 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 --2- 172.21.15.16:0/3997280382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32c07de60 0x7ff32c19dc60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 >> 172.21.15.16:0/3997280382 conn(0x7ff32c103110 msgr2=0x7ff32c103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 shutdown_connections 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:11.800+0000 7ff33136d700 1 -- 172.21.15.16:0/3997280382 wait complete. 2024-04-25T01:17:12.384 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-04-25T01:17:13.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:13 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:13.306+0000 7ff38ddf3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:17:13.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:13 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:13.482+0000 7ff38ddf3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:13.940 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:13 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:13.658+0000 7ff38ddf3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:17:13.940 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:13 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:13.955+0000 7ff38ddf3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:17:14.244 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:14 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:14.051+0000 7ff38ddf3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:17:14.245 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:14 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:14.260+0000 7ff38ddf3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:17:14.495 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:14 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:14.462+0000 7ff38ddf3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:15.086 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:14 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/160284992' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:17:15.364 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:15 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:15.102+0000 7ff38ddf3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.491 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.492 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:17:15.493 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:17:15.494 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:17:15.495 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:17:02.975509+0000", 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77de8d1700 1 Processor -- start 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77de8d1700 1 -- start start 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77de8d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d8106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77de8d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77d807bdd0 con 0x7f77d8106420 2024-04-25T01:17:15.496 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d8106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d8106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45690/0 (socket says 172.21.15.16:45690) 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.799+0000 7f77d7fff700 1 -- 172.21.15.16:0/3398070796 learned_addr learned my addr 172.21.15.16:0/3398070796 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.800+0000 7f77d7fff700 1 -- 172.21.15.16:0/3398070796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d8106d80 con 0x7f77d8106420 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.800+0000 7f77d7fff700 1 --2- 172.21.15.16:0/3398070796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d8106840 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f77c80098a0 tx=0x7f77c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4552c1a8e30681e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.800+0000 7f77d6ffd700 1 -- 172.21.15.16:0/3398070796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c8004160 con 0x7f77d8106420 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.800+0000 7f77d6ffd700 1 -- 172.21.15.16:0/3398070796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f77c8004300 con 0x7f77d8106420 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.800+0000 7f77d6ffd700 1 -- 172.21.15.16:0/3398070796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c8011a80 con 0x7f77d8106420 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 -- 172.21.15.16:0/3398070796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 msgr2=0x7f77d8106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 --2- 172.21.15.16:0/3398070796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d8106840 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f77c80098a0 tx=0x7f77c8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 -- 172.21.15.16:0/3398070796 shutdown_connections 2024-04-25T01:17:15.497 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 --2- 172.21.15.16:0/3398070796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d8106840 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 -- 172.21.15.16:0/3398070796 >> 172.21.15.16:0/3398070796 conn(0x7f77d8101fc0 msgr2=0x7f77d8104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 -- 172.21.15.16:0/3398070796 shutdown_connections 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.801+0000 7f77de8d1700 1 -- 172.21.15.16:0/3398070796 wait complete. 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77de8d1700 1 Processor -- start 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77de8d1700 1 -- start start 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77de8d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d819dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:15.498 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77de8d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77d8107db0 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d819dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d819dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45692/0 (socket says 172.21.15.16:45692) 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.802+0000 7f77d7fff700 1 -- 172.21.15.16:0/160284992 learned_addr learned my addr 172.21.15.16:0/160284992 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d7fff700 1 -- 172.21.15.16:0/160284992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d819e1b0 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d7fff700 1 --2- 172.21.15.16:0/160284992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d819dc70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77c8011a50 tx=0x7f77c8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c8003a80 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f77c8003c20 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d819e440 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c801a650 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d81a1130 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f77c802d3f0 con 0x7f77d8106420 2024-04-25T01:17:15.499 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.803+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f77c8019b30 con 0x7f77d8106420 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.804+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d8063890 con 0x7f77d8106420 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.806+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f77c8019d80 con 0x7f77d8106420 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.880+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f77d802dd70 con 0x7f77d8106420 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.880+0000 7f77d57fa700 1 -- 172.21.15.16:0/160284992 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f77c8032030 con 0x7f77d8106420 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 msgr2=0x7f77d819dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 --2- 172.21.15.16:0/160284992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d819dc70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77c8011a50 tx=0x7f77c8011b30 comp rx=0 tx=0).stop 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 shutdown_connections 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 --2- 172.21.15.16:0/160284992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77d8106420 0x7f77d819dc70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 >> 172.21.15.16:0/160284992 conn(0x7f77d8101fc0 msgr2=0x7f77d8194930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 shutdown_connections 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:14.881+0000 7f77de8d1700 1 -- 172.21.15.16:0/160284992 wait complete. 2024-04-25T01:17:15.500 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-04-25T01:17:15.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:15 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:15.544+0000 7ff38ddf3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:17:15.835 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:15 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:15.643+0000 7ff38ddf3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:17:15.835 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:15 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:15.850+0000 7ff38ddf3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:17:16.728 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:16.400+0000 7ff38ddf3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:17:16.728 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:16.494+0000 7ff38ddf3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:17:17.085 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:16.744+0000 7ff38ddf3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:17:17.085 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:16.882+0000 7ff38ddf3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:17:17.085 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:16.980+0000 7ff38ddf3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:17:17.085 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:17 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:17.100+0000 7ff38ddf3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:17.364 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:17 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:17.195+0000 7ff38ddf3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:17:18.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:18 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3074128006' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:17:18.267 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:18.060+0000 7ff38ddf3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:17:18.267 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:18.282+0000 7ff38ddf3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:17:18.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:18.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:18.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:17:18.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:17:18.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:17:18.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:17:18.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:17:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:17:18.608 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:17:02.975509+0000", 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.920+0000 7fddd1b02700 1 Processor -- start 2024-04-25T01:17:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.920+0000 7fddd1b02700 1 -- start start 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddd1b02700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc106800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddd1b02700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc07be50 con 0x7fddcc1063e0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45698/0 (socket says 172.21.15.16:45698) 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddcb7fe700 1 -- 172.21.15.16:0/50779420 learned_addr learned my addr 172.21.15.16:0/50779420 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddcb7fe700 1 -- 172.21.15.16:0/50779420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc106d40 con 0x7fddcc1063e0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddcb7fe700 1 --2- 172.21.15.16:0/50779420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc106800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fddb40098a0 tx=0x7fddb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d34c46371e999f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddca7fc700 1 -- 172.21.15.16:0/50779420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddb4004160 con 0x7fddcc1063e0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddca7fc700 1 -- 172.21.15.16:0/50779420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fddb4004300 con 0x7fddcc1063e0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.921+0000 7fddca7fc700 1 -- 172.21.15.16:0/50779420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddb4011a80 con 0x7fddcc1063e0 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 -- 172.21.15.16:0/50779420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 msgr2=0x7fddcc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 --2- 172.21.15.16:0/50779420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc106800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fddb40098a0 tx=0x7fddb4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:18.610 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 -- 172.21.15.16:0/50779420 shutdown_connections 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 --2- 172.21.15.16:0/50779420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc106800 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 -- 172.21.15.16:0/50779420 >> 172.21.15.16:0/50779420 conn(0x7fddcc102020 msgr2=0x7fddcc104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 -- 172.21.15.16:0/50779420 shutdown_connections 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.922+0000 7fddd1b02700 1 -- 172.21.15.16:0/50779420 wait complete. 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddd1b02700 1 Processor -- start 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddd1b02700 1 -- start start 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddd1b02700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc19dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddd1b02700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc107d70 con 0x7fddcc1063e0 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc19dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc19dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45706/0 (socket says 172.21.15.16:45706) 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddcb7fe700 1 -- 172.21.15.16:0/3074128006 learned_addr learned my addr 172.21.15.16:0/3074128006 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.923+0000 7fddcb7fe700 1 -- 172.21.15.16:0/3074128006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc19e1d0 con 0x7fddcc1063e0 2024-04-25T01:17:18.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddcb7fe700 1 --2- 172.21.15.16:0/3074128006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc19dc90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006af0 tx=0x7fddb4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddb40039e0 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fddb4003b80 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddcc19e460 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddb40196a0 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddcc1a1150 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fddb4019ce0 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.924+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fddb4024b60 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.925+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddcc063890 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:17.927+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fddb4043dd0 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.002+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fddcc02dd80 con 0x7fddcc1063e0 2024-04-25T01:17:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.002+0000 7fddc8ff9700 1 -- 172.21.15.16:0/3074128006 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fddb4032020 con 0x7fddcc1063e0 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 msgr2=0x7fddcc19dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 --2- 172.21.15.16:0/3074128006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc19dc90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006af0 tx=0x7fddb4011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 shutdown_connections 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 --2- 172.21.15.16:0/3074128006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddcc1063e0 0x7fddcc19dc90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 >> 172.21.15.16:0/3074128006 conn(0x7fddcc102020 msgr2=0x7fddcc1948f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 shutdown_connections 2024-04-25T01:17:18.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:18.004+0000 7fddd1b02700 1 -- 172.21.15.16:0/3074128006 wait complete. 2024-04-25T01:17:18.614 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (3/15)... 2024-04-25T01:17:19.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: Activating manager daemon x 2024-04-25T01:17:19.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: mgrmap e2: x(active, starting, since 0.00370625s) 2024-04-25T01:17:19.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:17:19.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: Manager daemon x is now available 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' 2024-04-25T01:17:19.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:19 smithi016 ceph-mon[29153]: from='mgr.14100 172.21.15.16:0/2091691381' entity='mgr.x' 2024-04-25T01:17:20.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:20 smithi016 ceph-mon[29153]: mgrmap e3: x(active, since 1.00749s) 2024-04-25T01:17:21.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:21 smithi016 ceph-mon[29153]: mgrmap e4: x(active, since 2s) 2024-04-25T01:17:21.994 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:17:21.995 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:17:21.997 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:17:21.997 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:17:22.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:17:22.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:17:22.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:17:02.975509+0000", 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.103+0000 7f6c9f3c2700 1 Processor -- start 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.103+0000 7f6c9f3c2700 1 -- start start 2024-04-25T01:17:22.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.103+0000 7f6c9f3c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9807c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.103+0000 7f6c9f3c2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9807c850 con 0x7f6c9807de20 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c9d15e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9807c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c9d15e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9807c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45786/0 (socket says 172.21.15.16:45786) 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c9d15e700 1 -- 172.21.15.16:0/206335562 learned_addr learned my addr 172.21.15.16:0/206335562 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c9d15e700 1 -- 172.21.15.16:0/206335562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9807c9d0 con 0x7f6c9807de20 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c9d15e700 1 --2- 172.21.15.16:0/206335562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9807c280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c940098a0 tx=0x7f6c94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f199d3760db04c11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c8ffff700 1 -- 172.21.15.16:0/206335562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c94004160 con 0x7f6c9807de20 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c8ffff700 1 -- 172.21.15.16:0/206335562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6c94004300 con 0x7f6c9807de20 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.104+0000 7f6c8ffff700 1 -- 172.21.15.16:0/206335562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c94011a80 con 0x7f6c9807de20 2024-04-25T01:17:22.010 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/206335562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 msgr2=0x7f6c9807c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 --2- 172.21.15.16:0/206335562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9807c280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c940098a0 tx=0x7f6c94009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/206335562 shutdown_connections 2024-04-25T01:17:22.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 --2- 172.21.15.16:0/206335562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9807c280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/206335562 >> 172.21.15.16:0/206335562 conn(0x7f6c98103180 msgr2=0x7f6c981055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.011 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/206335562 shutdown_connections 2024-04-25T01:17:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.105+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/206335562 wait complete. 2024-04-25T01:17:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9f3c2700 1 Processor -- start 2024-04-25T01:17:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9f3c2700 1 -- start start 2024-04-25T01:17:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9f3c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9f3c2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9810e170 con 0x7f6c9807de20 2024-04-25T01:17:22.012 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9d15e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.013 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9d15e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45788/0 (socket says 172.21.15.16:45788) 2024-04-25T01:17:22.013 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9d15e700 1 -- 172.21.15.16:0/631559078 learned_addr learned my addr 172.21.15.16:0/631559078 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:22.013 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9d15e700 1 -- 172.21.15.16:0/631559078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9819e1e0 con 0x7f6c9807de20 2024-04-25T01:17:22.013 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.106+0000 7f6c9d15e700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9819dca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6c94006af0 tx=0x7f6c94011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.013 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.107+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c940039e0 con 0x7f6c9807de20 2024-04-25T01:17:22.013 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.107+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6c94003b80 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.107+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9819e470 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.107+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c940196a0 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.107+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c981a1160 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.108+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f6c94019950 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.108+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c98063890 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.108+0000 7f6c8e7fc700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f6c84039000 0x7f6c8403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.108+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6c9404d730 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.108+0000 7f6c9c95d700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f6c84039000 0x7f6c8403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.108+0000 7f6c9c95d700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f6c84039000 0x7f6c8403b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6c88006fd0 tx=0x7f6c88006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.113+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c9404d150 con 0x7f6c9807de20 2024-04-25T01:17:22.014 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.410+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6c981a1330 con 0x7f6c9807de20 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.411+0000 7f6c8e7fc700 1 -- 172.21.15.16:0/631559078 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6c981a1330 con 0x7f6c9807de20 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f6c84039000 msgr2=0x7f6c8403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f6c84039000 0x7f6c8403b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6c88006fd0 tx=0x7f6c88006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 msgr2=0x7f6c9819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9819dca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6c94006af0 tx=0x7f6c94011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 shutdown_connections 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f6c84039000 0x7f6c8403b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 --2- 172.21.15.16:0/631559078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9807de20 0x7f6c9819dca0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 >> 172.21.15.16:0/631559078 conn(0x7f6c98103180 msgr2=0x7f6c98103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 shutdown_connections 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:21.413+0000 7f6c9f3c2700 1 -- 172.21.15.16:0/631559078 wait complete. 2024-04-25T01:17:22.015 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-04-25T01:17:22.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:22 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/631559078' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:17:23.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:23.182 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:17:23.183 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4adf9aa700 1 Processor -- start 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4adf9aa700 1 -- start start 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4adf9aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad8106890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4adf9aa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad807bdd0 con 0x7f4ad8106470 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4add746700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad8106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4add746700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad8106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38920/0 (socket says 172.21.15.16:38920) 2024-04-25T01:17:23.184 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4add746700 1 -- 172.21.15.16:0/3810533620 learned_addr learned my addr 172.21.15.16:0/3810533620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.412+0000 7f4add746700 1 -- 172.21.15.16:0/3810533620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad8106dd0 con 0x7f4ad8106470 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4add746700 1 --2- 172.21.15.16:0/3810533620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad8106890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4ad40098a0 tx=0x7f4ad4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38158b753c982d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4acffff700 1 -- 172.21.15.16:0/3810533620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad4004160 con 0x7f4ad8106470 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4acffff700 1 -- 172.21.15.16:0/3810533620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4ad4004300 con 0x7f4ad8106470 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4acffff700 1 -- 172.21.15.16:0/3810533620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad4011a80 con 0x7f4ad8106470 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4adf9aa700 1 -- 172.21.15.16:0/3810533620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 msgr2=0x7f4ad8106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4adf9aa700 1 --2- 172.21.15.16:0/3810533620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad8106890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4ad40098a0 tx=0x7f4ad4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4adf9aa700 1 -- 172.21.15.16:0/3810533620 shutdown_connections 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4adf9aa700 1 --2- 172.21.15.16:0/3810533620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad8106890 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.413+0000 7f4adf9aa700 1 -- 172.21.15.16:0/3810533620 >> 172.21.15.16:0/3810533620 conn(0x7f4ad8102050 msgr2=0x7f4ad8104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:23.185 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4adf9aa700 1 -- 172.21.15.16:0/3810533620 shutdown_connections 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4adf9aa700 1 -- 172.21.15.16:0/3810533620 wait complete. 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4adf9aa700 1 Processor -- start 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4adf9aa700 1 -- start start 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4adf9aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4adf9aa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad8107e00 con 0x7f4ad8106470 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4add746700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4add746700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad819dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38930/0 (socket says 172.21.15.16:38930) 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.414+0000 7f4add746700 1 -- 172.21.15.16:0/2976131145 learned_addr learned my addr 172.21.15.16:0/2976131145 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4add746700 1 -- 172.21.15.16:0/2976131145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad819e200 con 0x7f4ad8106470 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4add746700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad819dcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011a50 tx=0x7f4ad4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad4003a80 con 0x7f4ad8106470 2024-04-25T01:17:23.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4ad4003c20 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad819e490 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad401a650 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.415+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad81a1180 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.416+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f4ad4024ad0 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.416+0000 7f4ace7fc700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f4ac4039050 0x7f4ac403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.416+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4ad404d4c0 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.416+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad8063890 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.416+0000 7f4adcf45700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f4ac4039050 0x7f4ac403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.417+0000 7f4adcf45700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f4ac4039050 0x7f4ac403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.421+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ad400f2e0 con 0x7f4ad8106470 2024-04-25T01:17:23.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.626+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4ad802dcf0 con 0x7f4ad8106470 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.627+0000 7f4ace7fc700 1 -- 172.21.15.16:0/2976131145 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f4ad4017ca0 con 0x7f4ad8106470 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f4ac4039050 msgr2=0x7f4ac403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f4ac4039050 0x7f4ac403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 msgr2=0x7f4ad819dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad819dcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011a50 tx=0x7f4ad4011b30 comp rx=0 tx=0).stop 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 shutdown_connections 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f4ac4039050 0x7f4ac403b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 --2- 172.21.15.16:0/2976131145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad8106470 0x7f4ad819dcc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 >> 172.21.15.16:0/2976131145 conn(0x7f4ad8102050 msgr2=0x7f4ad81948e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 shutdown_connections 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:22.629+0000 7f4adf9aa700 1 -- 172.21.15.16:0/2976131145 wait complete. 2024-04-25T01:17:23.188 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-04-25T01:17:23.508 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:23 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2976131145' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:17:24.594 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:24 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3325018057' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T01:17:24.594 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: ignoring --setuser ceph since I am not root 2024-04-25T01:17:24.594 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: ignoring --setgroup ceph since I am not root 2024-04-25T01:17:24.595 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:24.468+0000 7f4494fb4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:17:24.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:24.658+0000 7f4494fb4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:24.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:24.764+0000 7f4494fb4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:17:24.886 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.634+0000 7fbeddb47700 1 Processor -- start 2024-04-25T01:17:24.886 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbeddb47700 1 -- start start 2024-04-25T01:17:24.886 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbeddb47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed810a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.886 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbeddb47700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbed807be50 con 0x7fbed81085f0 2024-04-25T01:17:24.886 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbed77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed810a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.886 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbed77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed810a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38942/0 (socket says 172.21.15.16:38942) 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbed77fe700 1 -- 172.21.15.16:0/3793482630 learned_addr learned my addr 172.21.15.16:0/3793482630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbed77fe700 1 -- 172.21.15.16:0/3793482630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed810af20 con 0x7fbed81085f0 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.635+0000 7fbed77fe700 1 --2- 172.21.15.16:0/3793482630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed810a9e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbec80098a0 tx=0x7fbec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff28056f3c96669 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbed67fc700 1 -- 172.21.15.16:0/3793482630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbec8004160 con 0x7fbed81085f0 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbed67fc700 1 -- 172.21.15.16:0/3793482630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fbec8004300 con 0x7fbed81085f0 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbed67fc700 1 -- 172.21.15.16:0/3793482630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbec8011a80 con 0x7fbed81085f0 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 -- 172.21.15.16:0/3793482630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 msgr2=0x7fbed810a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 --2- 172.21.15.16:0/3793482630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed810a9e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbec80098a0 tx=0x7fbec8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 -- 172.21.15.16:0/3793482630 shutdown_connections 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 --2- 172.21.15.16:0/3793482630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed810a9e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 -- 172.21.15.16:0/3793482630 >> 172.21.15.16:0/3793482630 conn(0x7fbed8102020 msgr2=0x7fbed8104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 -- 172.21.15.16:0/3793482630 shutdown_connections 2024-04-25T01:17:24.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.636+0000 7fbeddb47700 1 -- 172.21.15.16:0/3793482630 wait complete. 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbeddb47700 1 Processor -- start 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbeddb47700 1 -- start start 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbeddb47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed819dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbeddb47700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbed810bef0 con 0x7fbed81085f0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbed77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed819dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbed77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed819dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38944/0 (socket says 172.21.15.16:38944) 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.637+0000 7fbed77fe700 1 -- 172.21.15.16:0/3325018057 learned_addr learned my addr 172.21.15.16:0/3325018057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbed77fe700 1 -- 172.21.15.16:0/3325018057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed819e240 con 0x7fbed81085f0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbed77fe700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed819dd00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbec8011a50 tx=0x7fbec8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbec8003a80 con 0x7fbed81085f0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed819e4d0 con 0x7fbed81085f0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fbec8003c20 con 0x7fbed81085f0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbec801a650 con 0x7fbed81085f0 2024-04-25T01:17:24.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.638+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed81a1170 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.639+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fbec801a970 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.639+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed8063890 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.639+0000 7fbed4ff9700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7fbec0039000 0x7fbec003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.639+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbec804d4d0 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.639+0000 7fbed6ffd700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7fbec0039000 0x7fbec003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.639+0000 7fbed6ffd700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7fbec0039000 0x7fbec003b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbecc006fd0 tx=0x7fbecc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.644+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbec8019c70 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:23.910+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fbed81a1340 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.295+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fbed81a1340 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.297+0000 7fbed4ff9700 1 -- 172.21.15.16:0/3325018057 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fbec801ad10 con 0x7fbed81085f0 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7fbec0039000 msgr2=0x7fbec003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7fbec0039000 0x7fbec003b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbecc006fd0 tx=0x7fbecc006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 msgr2=0x7fbed819dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed819dd00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbec8011a50 tx=0x7fbec8011b30 comp rx=0 tx=0).stop 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 shutdown_connections 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7fbec0039000 0x7fbec003b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 --2- 172.21.15.16:0/3325018057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbed81085f0 0x7fbed819dd00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 >> 172.21.15.16:0/3325018057 conn(0x7fbed8102020 msgr2=0x7fbed8102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 shutdown_connections 2024-04-25T01:17:24.890 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:24.298+0000 7fbeddb47700 1 -- 172.21.15.16:0/3325018057 wait complete. 2024-04-25T01:17:25.170 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:25 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:25.097+0000 7f4494fb4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:17:25.171 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:25 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:25.186+0000 7f4494fb4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:17:25.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:25 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3325018057' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T01:17:25.506 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:25 smithi016 ceph-mon[29153]: mgrmap e5: x(active, since 6s) 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.316+0000 7f018bb3d700 1 Processor -- start 2024-04-25T01:17:26.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.316+0000 7f018bb3d700 1 -- start start 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f018bb3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f0184106e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f018bb3d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0184107460 con 0x7f0184106a70 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f01898d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f0184106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f01898d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f0184106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38960/0 (socket says 172.21.15.16:38960) 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f01898d9700 1 -- 172.21.15.16:0/828869866 learned_addr learned my addr 172.21.15.16:0/828869866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f01898d9700 1 -- 172.21.15.16:0/828869866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0184107c70 con 0x7f0184106a70 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.317+0000 7f01898d9700 1 --2- 172.21.15.16:0/828869866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f0184106e90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f01800098a0 tx=0x7f0180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f47e63ff5a7093 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f01888d7700 1 -- 172.21.15.16:0/828869866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0180004160 con 0x7f0184106a70 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f01888d7700 1 -- 172.21.15.16:0/828869866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0180004300 con 0x7f0184106a70 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f01888d7700 1 -- 172.21.15.16:0/828869866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0180011a80 con 0x7f0184106a70 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 -- 172.21.15.16:0/828869866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 msgr2=0x7f0184106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 --2- 172.21.15.16:0/828869866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f0184106e90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f01800098a0 tx=0x7f0180009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 -- 172.21.15.16:0/828869866 shutdown_connections 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 --2- 172.21.15.16:0/828869866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f0184106e90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 -- 172.21.15.16:0/828869866 >> 172.21.15.16:0/828869866 conn(0x7f0184102010 msgr2=0x7f0184104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 -- 172.21.15.16:0/828869866 shutdown_connections 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.318+0000 7f018bb3d700 1 -- 172.21.15.16:0/828869866 wait complete. 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f018bb3d700 1 Processor -- start 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f018bb3d700 1 -- start start 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f018bb3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f018407ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f018bb3d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f018410b020 con 0x7f0184106a70 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f01898d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f018407ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f01898d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f018407ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38970/0 (socket says 172.21.15.16:38970) 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.319+0000 7f01898d9700 1 -- 172.21.15.16:0/1241695470 learned_addr learned my addr 172.21.15.16:0/1241695470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f01898d9700 1 -- 172.21.15.16:0/1241695470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018419e7f0 con 0x7f0184106a70 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f01898d9700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f018407ba50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0180000c00 tx=0x7f0180011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01800039f0 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018419e9c0 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0180003b90 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f018001a740 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.320+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018419ede0 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.321+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f018001aa40 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.321+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0184063890 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.321+0000 7f017affd700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 0x7f017003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.321+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f018004d430 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.321+0000 7f01890d8700 1 -- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 msgr2=0x7f017003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.321+0000 7f01890d8700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 0x7f017003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.326+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0180020240 con 0x7f0184106a70 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.522+0000 7f01890d8700 1 -- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 msgr2=0x7f017003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:26.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.522+0000 7f01890d8700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 0x7f017003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.628+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f018419f660 con 0x7f0184106a70 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.629+0000 7f017affd700 1 -- 172.21.15.16:0/1241695470 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f018000ba00 con 0x7f0184106a70 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.633+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 msgr2=0x7f017003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.633+0000 7f018bb3d700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 0x7f017003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.633+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 msgr2=0x7f018407ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.633+0000 7f018bb3d700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f018407ba50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0180000c00 tx=0x7f0180011b00 comp rx=0 tx=0).stop 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.633+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 shutdown_connections 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.634+0000 7f018bb3d700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f0170039050 0x7f017003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.634+0000 7f018bb3d700 1 --2- 172.21.15.16:0/1241695470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0184106a70 0x7f018407ba50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.634+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 >> 172.21.15.16:0/1241695470 conn(0x7f0184102010 msgr2=0x7f0184102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.634+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 shutdown_connections 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:25.634+0000 7f018bb3d700 1 -- 172.21.15.16:0/1241695470 wait complete. 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-25T01:17:26.447 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-04-25T01:17:26.561 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:26 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1241695470' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:17:26.562 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:26.434+0000 7f4494fb4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:17:26.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:26.615+0000 7f4494fb4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:26.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:26.791+0000 7f4494fb4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:17:27.174 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:27.095+0000 7f4494fb4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:17:27.174 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:27.190+0000 7f4494fb4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:17:27.499 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:27.398+0000 7f4494fb4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:17:27.864 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:27.597+0000 7f4494fb4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:28.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:28.217+0000 7f4494fb4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:17:28.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:28.652+0000 7f4494fb4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:17:28.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:28.751+0000 7f4494fb4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:17:28.945 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:28.961+0000 7f4494fb4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:17:29.861 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:29.525+0000 7f4494fb4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:17:29.861 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:29.621+0000 7f4494fb4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:17:30.114 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:29.874+0000 7f4494fb4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:17:30.114 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:30.013+0000 7f4494fb4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:17:30.115 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:30.112+0000 7f4494fb4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:17:30.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:30.232+0000 7f4494fb4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:30.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:30.328+0000 7f4494fb4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:17:31.427 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:31.167+0000 7f4494fb4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:17:31.427 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:31.390+0000 7f4494fb4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:17:31.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: Active manager daemon x restarted 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: Activating manager daemon x 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: mgrmap e6: x(active, starting, since 0.00342323s) 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: Manager daemon x is now available 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:31.706 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:31.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:31.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:17:31.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:31 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.906+0000 7f5238edc700 1 Processor -- start 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f5238edc700 1 -- start start 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f5238edc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f5234106870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f5238edc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f523407bdb0 con 0x7f5234106450 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f523259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f5234106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f523259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f5234106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38974/0 (socket says 172.21.15.16:38974) 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f523259c700 1 -- 172.21.15.16:0/65839034 learned_addr learned my addr 172.21.15.16:0/65839034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f523259c700 1 -- 172.21.15.16:0/65839034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5234106db0 con 0x7f5234106450 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.907+0000 7f523259c700 1 --2- 172.21.15.16:0/65839034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f5234106870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52280098a0 tx=0x7f5228009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83d1aa3864e27ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.908+0000 7f523159a700 1 -- 172.21.15.16:0/65839034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5228004160 con 0x7f5234106450 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.908+0000 7f523159a700 1 -- 172.21.15.16:0/65839034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5228004300 con 0x7f5234106450 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.908+0000 7f523159a700 1 -- 172.21.15.16:0/65839034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5228011a80 con 0x7f5234106450 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.908+0000 7f5238edc700 1 -- 172.21.15.16:0/65839034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 msgr2=0x7f5234106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.909+0000 7f5238edc700 1 --2- 172.21.15.16:0/65839034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f5234106870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52280098a0 tx=0x7f5228009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.909+0000 7f5238edc700 1 -- 172.21.15.16:0/65839034 shutdown_connections 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.909+0000 7f5238edc700 1 --2- 172.21.15.16:0/65839034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f5234106870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.909+0000 7f5238edc700 1 -- 172.21.15.16:0/65839034 >> 172.21.15.16:0/65839034 conn(0x7f5234101ff0 msgr2=0x7f5234104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.020 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.909+0000 7f5238edc700 1 -- 172.21.15.16:0/65839034 shutdown_connections 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.909+0000 7f5238edc700 1 -- 172.21.15.16:0/65839034 wait complete. 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f5238edc700 1 Processor -- start 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f5238edc700 1 -- start start 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f5238edc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f523419dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f5238edc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5234107de0 con 0x7f5234106450 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f523259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f523419dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f523259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f523419dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38980/0 (socket says 172.21.15.16:38980) 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f523259c700 1 -- 172.21.15.16:0/1595525434 learned_addr learned my addr 172.21.15.16:0/1595525434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.910+0000 7f523259c700 1 -- 172.21.15.16:0/1595525434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523419e1a0 con 0x7f5234106450 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.911+0000 7f523259c700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f523419dc60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5228006af0 tx=0x7f5228011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.911+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52280039e0 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.911+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5228003b80 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.911+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523419e430 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.911+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52341a1120 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.911+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52280196a0 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.912+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f5228019990 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.912+0000 7f52237fe700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.912+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 --> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f521c03bc20 con 0x7f521c039050 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.912+0000 7f5231d9b700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 msgr2=0x7f521c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.912+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f522804e1e0 con 0x7f5234106450 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:26.912+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:27.113+0000 7f5231d9b700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 msgr2=0x7f521c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:27.113+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:27.514+0000 7f5231d9b700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 msgr2=0x7f521c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:27.514+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:28.315+0000 7f5231d9b700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 msgr2=0x7f521c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:28.315+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:29.916+0000 7f5231d9b700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 msgr2=0x7f521c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1662737541 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:29.916+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:31.394+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f5228019cd0 con 0x7f5234106450 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:31.394+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 msgr2=0x7f521c03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:31.394+0000 7f52237fe700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.395+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f5228017920 con 0x7f5234106450 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.396+0000 7f52237fe700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f521c03c980 0x7f521c03ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.396+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f521c03bc20 con 0x7f521c03c980 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.396+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f521c03c980 0x7f521c03ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.397+0000 7f5231d9b700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f521c03c980 0x7f521c03ed70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5224003a10 tx=0x7f52240092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.398+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f521c03bc20 con 0x7f521c03c980 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.404+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5214002630 con 0x7f521c03c980 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f52237fe700 1 -- 172.21.15.16:0/1595525434 <== mgr.14120 v2:172.21.15.16:6800/1190188268 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f5214002630 con 0x7f521c03c980 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f521c03c980 msgr2=0x7f521c03ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f521c03c980 0x7f521c03ed70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5224003a10 tx=0x7f52240092b0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 msgr2=0x7f523419dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f523419dc60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5228006af0 tx=0x7f5228011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 shutdown_connections 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f521c03c980 0x7f521c03ed70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5234106450 0x7f523419dc60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 --2- 172.21.15.16:0/1595525434 >> [v2:172.21.15.16:6800/1662737541,v1:172.21.15.16:6801/1662737541] conn(0x7f521c039050 0x7f521c03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 >> 172.21.15.16:0/1595525434 conn(0x7f5234101ff0 msgr2=0x7f52341948c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 shutdown_connections 2024-04-25T01:17:33.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:32.405+0000 7f5238edc700 1 -- 172.21.15.16:0/1595525434 wait complete. 2024-04-25T01:17:33.025 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-04-25T01:17:33.025 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-04-25T01:17:33.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:33 smithi016 ceph-mon[29153]: Found migration_current of "None". Setting to last migration. 2024-04-25T01:17:33.328 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:33 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:33.328 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:33 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:33.328 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:33 smithi016 ceph-mon[29153]: mgrmap e7: x(active, since 1.00595s) 2024-04-25T01:17:34.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:34 smithi016 ceph-mon[29153]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:17:34.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:34 smithi016 ceph-mon[29153]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:17:34.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:34 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:34.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:34 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:34.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:34 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:34.285 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.448+0000 7fa55d506700 1 Processor -- start 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.448+0000 7fa55d506700 1 -- start start 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.448+0000 7fa55d506700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa55810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.448+0000 7fa55d506700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa55807be50 con 0x7fa558108600 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa556ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa55810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa556ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa55810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51628/0 (socket says 172.21.15.16:51628) 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa556ffd700 1 -- 172.21.15.16:0/3523211510 learned_addr learned my addr 172.21.15.16:0/3523211510 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa556ffd700 1 -- 172.21.15.16:0/3523211510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55810af30 con 0x7fa558108600 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa556ffd700 1 --2- 172.21.15.16:0/3523211510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa55810a9f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e80 tx=0x7fa54800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e144ac494206e14e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa555ffb700 1 -- 172.21.15.16:0/3523211510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa548017070 con 0x7fa558108600 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa555ffb700 1 -- 172.21.15.16:0/3523211510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa54800b8c0 con 0x7fa558108600 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.449+0000 7fa555ffb700 1 -- 172.21.15.16:0/3523211510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa548004520 con 0x7fa558108600 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 -- 172.21.15.16:0/3523211510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 msgr2=0x7fa55810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 --2- 172.21.15.16:0/3523211510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa55810a9f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e80 tx=0x7fa54800b330 comp rx=0 tx=0).stop 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 -- 172.21.15.16:0/3523211510 shutdown_connections 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 --2- 172.21.15.16:0/3523211510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa55810a9f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 -- 172.21.15.16:0/3523211510 >> 172.21.15.16:0/3523211510 conn(0x7fa558102050 msgr2=0x7fa558104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 -- 172.21.15.16:0/3523211510 shutdown_connections 2024-04-25T01:17:34.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 -- 172.21.15.16:0/3523211510 wait complete. 2024-04-25T01:17:34.296 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.450+0000 7fa55d506700 1 Processor -- start 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa55d506700 1 -- start start 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa55d506700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa558107e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa55d506700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa55810bf00 con 0x7fa558108600 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa556ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa558107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa556ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa558107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51634/0 (socket says 172.21.15.16:51634) 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa556ffd700 1 -- 172.21.15.16:0/279080441 learned_addr learned my addr 172.21.15.16:0/279080441 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:34.297 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa556ffd700 1 -- 172.21.15.16:0/279080441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5581083c0 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.451+0000 7fa556ffd700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa558107e80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e50 tx=0x7fa548004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.452+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa548020020 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.452+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa54800bba0 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.452+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5581065b0 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.452+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa548017040 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.452+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa558106af0 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.452+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fa548023020 con 0x7fa558108600 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.453+0000 7fa53ffff700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fa540038ed0 0x7fa54003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.453+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa54800f050 con 0x7fa558108600 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.453+0000 7fa5567fc700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fa540038ed0 0x7fa54003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.453+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa558063890 con 0x7fa558108600 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.453+0000 7fa5567fc700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fa540038ed0 0x7fa54003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006fd0 tx=0x7fa54c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.458+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa54802a020 con 0x7fa558108600 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.682+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fa558107370 con 0x7fa540038ed0 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.687+0000 7fa53ffff700 1 -- 172.21.15.16:0/279080441 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa558107370 con 0x7fa540038ed0 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fa540038ed0 msgr2=0x7fa54003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fa540038ed0 0x7fa54003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006fd0 tx=0x7fa54c006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 msgr2=0x7fa558107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa558107e80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa548009e50 tx=0x7fa548004670 comp rx=0 tx=0).stop 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 shutdown_connections 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fa540038ed0 0x7fa54003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 --2- 172.21.15.16:0/279080441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa558108600 0x7fa558107e80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 >> 172.21.15.16:0/279080441 conn(0x7fa558102050 msgr2=0x7fa558102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 shutdown_connections 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:33.689+0000 7fa55d506700 1 -- 172.21.15.16:0/279080441 wait complete. 2024-04-25T01:17:35.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:35 smithi016 ceph-mon[29153]: [25/Apr/2024:01:17:33] ENGINE Bus STARTING 2024-04-25T01:17:35.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:35 smithi016 ceph-mon[29153]: [25/Apr/2024:01:17:33] ENGINE Serving on https://172.21.15.16:7150 2024-04-25T01:17:35.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:35 smithi016 ceph-mon[29153]: [25/Apr/2024:01:17:33] ENGINE Serving on http://172.21.15.16:8765 2024-04-25T01:17:35.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:35 smithi016 ceph-mon[29153]: [25/Apr/2024:01:17:33] ENGINE Bus STARTED 2024-04-25T01:17:35.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:35 smithi016 ceph-mon[29153]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:35.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:35 smithi016 ceph-mon[29153]: mgrmap e8: x(active, since 2s) 2024-04-25T01:17:35.564 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T01:17:35.564 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.790+0000 7f9259349700 1 Processor -- start 2024-04-25T01:17:35.564 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9259349700 1 -- start start 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9259349700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f925410aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9259349700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f925407be50 con 0x7f9254108630 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9252ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f925410aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9252ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f925410aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51640/0 (socket says 172.21.15.16:51640) 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9252ffd700 1 -- 172.21.15.16:0/1963824043 learned_addr learned my addr 172.21.15.16:0/1963824043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9252ffd700 1 -- 172.21.15.16:0/1963824043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925410af60 con 0x7f9254108630 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.791+0000 7f9252ffd700 1 --2- 172.21.15.16:0/1963824043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f925410aa20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92440098a0 tx=0x7f9244009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa7635d2e3eb55f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9251ffb700 1 -- 172.21.15.16:0/1963824043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9244004160 con 0x7f9254108630 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9251ffb700 1 -- 172.21.15.16:0/1963824043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9244004300 con 0x7f9254108630 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9251ffb700 1 -- 172.21.15.16:0/1963824043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9244011a80 con 0x7f9254108630 2024-04-25T01:17:35.565 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 -- 172.21.15.16:0/1963824043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 msgr2=0x7f925410aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 --2- 172.21.15.16:0/1963824043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f925410aa20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92440098a0 tx=0x7f9244009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 -- 172.21.15.16:0/1963824043 shutdown_connections 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 --2- 172.21.15.16:0/1963824043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f925410aa20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 -- 172.21.15.16:0/1963824043 >> 172.21.15.16:0/1963824043 conn(0x7f92541020a0 msgr2=0x7f92541044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 -- 172.21.15.16:0/1963824043 shutdown_connections 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.792+0000 7f9259349700 1 -- 172.21.15.16:0/1963824043 wait complete. 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.793+0000 7f9259349700 1 Processor -- start 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.793+0000 7f9259349700 1 -- start start 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.793+0000 7f9259349700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f9254107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.793+0000 7f9259349700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f925410bf30 con 0x7f9254108630 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9252ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f9254107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9252ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f9254107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51654/0 (socket says 172.21.15.16:51654) 2024-04-25T01:17:35.566 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9252ffd700 1 -- 172.21.15.16:0/358400590 learned_addr learned my addr 172.21.15.16:0/358400590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9252ffd700 1 -- 172.21.15.16:0/358400590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92541083f0 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9252ffd700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f9254107eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9244011a50 tx=0x7f9244011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9244003a80 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9244003c20 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92541065e0 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f924401a650 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.794+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9254106b20 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.795+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9254063890 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.795+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f924401a920 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.795+0000 7f923bfff700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f923c0390a0 0x7f923c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.795+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f924404d540 con 0x7f9254108630 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.796+0000 7f92527fc700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f923c0390a0 0x7f923c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.796+0000 7f92527fc700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f923c0390a0 0x7f923c03b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9248006fd0 tx=0x7f9248006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.567 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:34.801+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9244024b80 con 0x7f9254108630 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.009+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9254107380 con 0x7f923c0390a0 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.010+0000 7f923bfff700 1 -- 172.21.15.16:0/358400590 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9254107380 con 0x7f923c0390a0 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f923c0390a0 msgr2=0x7f923c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f923c0390a0 0x7f923c03b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9248006fd0 tx=0x7f9248006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 msgr2=0x7f9254107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f9254107eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9244011a50 tx=0x7f9244011b30 comp rx=0 tx=0).stop 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 shutdown_connections 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f923c0390a0 0x7f923c03b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 --2- 172.21.15.16:0/358400590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9254108630 0x7f9254107eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 >> 172.21.15.16:0/358400590 conn(0x7f92541020a0 msgr2=0x7f9254102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 shutdown_connections 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:35.012+0000 7f9259349700 1 -- 172.21.15.16:0/358400590 wait complete. 2024-04-25T01:17:35.568 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-04-25T01:17:36.456 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-mon[29153]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: Generating public/private rsa key pair. 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: Your identification has been saved in /tmp/tmp33w5kscz/key. 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: Your public key has been saved in /tmp/tmp33w5kscz/key.pub. 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: The key fingerprint is: 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: SHA256:IxFaWhsF0SuKeXayBXl85RIVMDWohrjxOt7Sv5Ic9jo ceph-605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: The key's randomart image is: 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: +---[RSA 3072]----+ 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | B=+==. | 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | = ++... | 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | .o+o. = | 2024-04-25T01:17:36.770 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | o + *.+ . | 2024-04-25T01:17:36.771 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | * =.oS. | 2024-04-25T01:17:36.771 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | + O o. . | 2024-04-25T01:17:36.771 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | B O | 2024-04-25T01:17:36.771 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | +.E . | 2024-04-25T01:17:36.771 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: | ..oo=o. | 2024-04-25T01:17:36.771 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: +----[SHA256]-----+ 2024-04-25T01:17:37.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c71ee700 1 Processor -- start 2024-04-25T01:17:37.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c71ee700 1 -- start start 2024-04-25T01:17:37.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c71ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c0107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c71ee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61c0108590 con 0x7f61c0107ba0 2024-04-25T01:17:37.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c4f8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c0107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c4f8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c0107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51664/0 (socket says 172.21.15.16:51664) 2024-04-25T01:17:37.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.007+0000 7f61c4f8a700 1 -- 172.21.15.16:0/2054321212 learned_addr learned my addr 172.21.15.16:0/2054321212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:37.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.008+0000 7f61c4f8a700 1 -- 172.21.15.16:0/2054321212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c0108da0 con 0x7f61c0107ba0 2024-04-25T01:17:37.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.008+0000 7f61c4f8a700 1 --2- 172.21.15.16:0/2054321212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c0107fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61b40098a0 tx=0x7f61b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa0f1c8ae402c7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.008+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2054321212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61b4004160 con 0x7f61c0107ba0 2024-04-25T01:17:37.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.008+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2054321212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f61b4004300 con 0x7f61c0107ba0 2024-04-25T01:17:37.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.008+0000 7f61bf7fe700 1 -- 172.21.15.16:0/2054321212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61b4011a80 con 0x7f61c0107ba0 2024-04-25T01:17:37.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 -- 172.21.15.16:0/2054321212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 msgr2=0x7f61c0107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 --2- 172.21.15.16:0/2054321212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c0107fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61b40098a0 tx=0x7f61b4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 -- 172.21.15.16:0/2054321212 shutdown_connections 2024-04-25T01:17:37.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 --2- 172.21.15.16:0/2054321212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c0107fc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 -- 172.21.15.16:0/2054321212 >> 172.21.15.16:0/2054321212 conn(0x7f61c0103180 msgr2=0x7f61c01055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 -- 172.21.15.16:0/2054321212 shutdown_connections 2024-04-25T01:17:37.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 -- 172.21.15.16:0/2054321212 wait complete. 2024-04-25T01:17:37.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 Processor -- start 2024-04-25T01:17:37.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.009+0000 7f61c71ee700 1 -- start start 2024-04-25T01:17:37.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c71ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c019baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c71ee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61c010c150 con 0x7f61c0107ba0 2024-04-25T01:17:37.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c4f8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c019baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c4f8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c019baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51666/0 (socket says 172.21.15.16:51666) 2024-04-25T01:17:37.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c4f8a700 1 -- 172.21.15.16:0/2139138335 learned_addr learned my addr 172.21.15.16:0/2139138335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:37.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c4f8a700 1 -- 172.21.15.16:0/2139138335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c019c030 con 0x7f61c0107ba0 2024-04-25T01:17:37.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c4f8a700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c019baf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006af0 tx=0x7f61b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61b40039e0 con 0x7f61c0107ba0 2024-04-25T01:17:37.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f61b4003b80 con 0x7f61c0107ba0 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61b40196a0 con 0x7f61c0107ba0 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c019c2c0 con 0x7f61c0107ba0 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.010+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c019efb0 con 0x7f61c0107ba0 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.011+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f61b401ab50 con 0x7f61c0107ba0 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.012+0000 7f61bdffb700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f61b0039050 0x7f61b003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.012+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c0063890 con 0x7f61c0107ba0 2024-04-25T01:17:37.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.012+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f61b404d520 con 0x7f61c0107ba0 2024-04-25T01:17:37.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.012+0000 7f61bffff700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f61b0039050 0x7f61b003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.012+0000 7f61bffff700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f61b0039050 0x7f61b003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f61ac006fd0 tx=0x7f61ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.017+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f61b400b3e0 con 0x7f61c0107ba0 2024-04-25T01:17:37.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.226+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f61c0104380 con 0x7f61b0039050 2024-04-25T01:17:37.070 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.478+0000 7f61bdffb700 1 -- 172.21.15.16:0/2139138335 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f61c0104380 con 0x7f61b0039050 2024-04-25T01:17:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f61b0039050 msgr2=0x7f61b003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f61b0039050 0x7f61b003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f61ac006fd0 tx=0x7f61ac006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 msgr2=0x7f61c019baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c019baf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006af0 tx=0x7f61b4011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 shutdown_connections 2024-04-25T01:17:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f61b0039050 0x7f61b003b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 --2- 172.21.15.16:0/2139138335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c0107ba0 0x7f61c019baf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 >> 172.21.15.16:0/2139138335 conn(0x7f61c0103180 msgr2=0x7f61c0103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.480+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 shutdown_connections 2024-04-25T01:17:37.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:36.481+0000 7f61c71ee700 1 -- 172.21.15.16:0/2139138335 wait complete. 2024-04-25T01:17:37.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:37 smithi016 ceph-mon[29153]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:37.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:37 smithi016 ceph-mon[29153]: Generating ssh key... 2024-04-25T01:17:37.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:37 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:37.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:37 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.532+0000 7f8cd60fd700 1 Processor -- start 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.532+0000 7f8cd60fd700 1 -- start start 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.532+0000 7f8cd60fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd0107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.532+0000 7f8cd60fd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd0108540 con 0x7f8cd0107b50 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8ccf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd0107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8ccf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd0107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51668/0 (socket says 172.21.15.16:51668) 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8ccf7fe700 1 -- 172.21.15.16:0/3966304258 learned_addr learned my addr 172.21.15.16:0/3966304258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8ccf7fe700 1 -- 172.21.15.16:0/3966304258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd0108d50 con 0x7f8cd0107b50 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8ccf7fe700 1 --2- 172.21.15.16:0/3966304258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd0107f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00098a0 tx=0x7f8cc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782f1a2758018971 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8cce7fc700 1 -- 172.21.15.16:0/3966304258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc0004160 con 0x7f8cd0107b50 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8cce7fc700 1 -- 172.21.15.16:0/3966304258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8cc0004300 con 0x7f8cd0107b50 2024-04-25T01:17:38.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.533+0000 7f8cce7fc700 1 -- 172.21.15.16:0/3966304258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc0011a80 con 0x7f8cd0107b50 2024-04-25T01:17:38.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3966304258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 msgr2=0x7f8cd0107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 --2- 172.21.15.16:0/3966304258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd0107f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00098a0 tx=0x7f8cc0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3966304258 shutdown_connections 2024-04-25T01:17:38.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 --2- 172.21.15.16:0/3966304258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd0107f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3966304258 >> 172.21.15.16:0/3966304258 conn(0x7f8cd0103110 msgr2=0x7f8cd0105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3966304258 shutdown_connections 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3966304258 wait complete. 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.534+0000 7f8cd60fd700 1 Processor -- start 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8cd60fd700 1 -- start start 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8cd60fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd007db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8cd60fd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd010c100 con 0x7f8cd0107b50 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8ccf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd007db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8ccf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd007db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51678/0 (socket says 172.21.15.16:51678) 2024-04-25T01:17:38.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8ccf7fe700 1 -- 172.21.15.16:0/3222052126 learned_addr learned my addr 172.21.15.16:0/3222052126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8ccf7fe700 1 -- 172.21.15.16:0/3222052126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd007e050 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.535+0000 7f8ccf7fe700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd007db10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0006af0 tx=0x7f8cc0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.536+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc00039e0 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.536+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8cc0003b80 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.536+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd007c240 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.536+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc00196a0 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.536+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd007c780 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.537+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8cc001ab50 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.537+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd0063890 con 0x7f8cd0107b50 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.537+0000 7f8cccff9700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f8cb8039050 0x7f8cb803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.537+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8cc004d570 con 0x7f8cd0107b50 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.537+0000 7f8cceffd700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f8cb8039050 0x7f8cb803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.537+0000 7f8cceffd700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f8cb8039050 0x7f8cb803b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4006fd0 tx=0x7f8cc4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.542+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8cc000b3e0 con 0x7f8cd0107b50 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.750+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8cd007cf90 con 0x7f8cb8039050 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.750+0000 7f8cccff9700 1 -- 172.21.15.16:0/3222052126 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f8cd007cf90 con 0x7f8cb8039050 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f8cb8039050 msgr2=0x7f8cb803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f8cb8039050 0x7f8cb803b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4006fd0 tx=0x7f8cc4006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 msgr2=0x7f8cd007db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd007db10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0006af0 tx=0x7f8cc0011c40 comp rx=0 tx=0).stop 2024-04-25T01:17:38.383 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 shutdown_connections 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f8cb8039050 0x7f8cb803b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 --2- 172.21.15.16:0/3222052126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cd0107b50 0x7f8cd007db10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.752+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 >> 172.21.15.16:0/3222052126 conn(0x7f8cd0103110 msgr2=0x7f8cd0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.753+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 shutdown_connections 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:37.753+0000 7f8cd60fd700 1 -- 172.21.15.16:0/3222052126 wait complete. 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T01:17:38.384 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-04-25T01:17:38.625 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:38 smithi016 ceph-mon[29153]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:39.673 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:39 smithi016 ceph-mon[29153]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:40 smithi016 ceph-mon[29153]: Deploying cephadm binary to smithi016 2024-04-25T01:17:44.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-04-25T01:17:44.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.852+0000 7f400bf97700 1 Processor -- start 2024-04-25T01:17:44.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f400bf97700 1 -- start start 2024-04-25T01:17:44.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f400bf97700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f400410aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f400bf97700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f400407bdd0 con 0x7f4004108680 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f4009d33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f400410aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f4009d33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f400410aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51692/0 (socket says 172.21.15.16:51692) 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f4009d33700 1 -- 172.21.15.16:0/865655630 learned_addr learned my addr 172.21.15.16:0/865655630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.853+0000 7f4009d33700 1 -- 172.21.15.16:0/865655630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400410afb0 con 0x7f4004108680 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f4009d33700 1 --2- 172.21.15.16:0/865655630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f400410aa70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f40000098a0 tx=0x7f4000009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfe5b0a6a1e061c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f4008d31700 1 -- 172.21.15.16:0/865655630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4000004160 con 0x7f4004108680 2024-04-25T01:17:44.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f4008d31700 1 -- 172.21.15.16:0/865655630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4000004300 con 0x7f4004108680 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f4008d31700 1 -- 172.21.15.16:0/865655630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4000011a80 con 0x7f4004108680 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f400bf97700 1 -- 172.21.15.16:0/865655630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 msgr2=0x7f400410aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f400bf97700 1 --2- 172.21.15.16:0/865655630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f400410aa70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f40000098a0 tx=0x7f4000009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f400bf97700 1 -- 172.21.15.16:0/865655630 shutdown_connections 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f400bf97700 1 --2- 172.21.15.16:0/865655630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f400410aa70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.854+0000 7f400bf97700 1 -- 172.21.15.16:0/865655630 >> 172.21.15.16:0/865655630 conn(0x7f4004102050 msgr2=0x7f4004104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.855+0000 7f400bf97700 1 -- 172.21.15.16:0/865655630 shutdown_connections 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.855+0000 7f400bf97700 1 -- 172.21.15.16:0/865655630 wait complete. 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.855+0000 7f400bf97700 1 Processor -- start 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.855+0000 7f400bf97700 1 -- start start 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.855+0000 7f400bf97700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f4004107f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.855+0000 7f400bf97700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f400410bf10 con 0x7f4004108680 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f4009d33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f4004107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f4009d33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f4004107f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51700/0 (socket says 172.21.15.16:51700) 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f4009d33700 1 -- 172.21.15.16:0/988892930 learned_addr learned my addr 172.21.15.16:0/988892930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f4009d33700 1 -- 172.21.15.16:0/988892930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4004108440 con 0x7f4004108680 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f4009d33700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f4004107f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4000011a50 tx=0x7f4000011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4000003a80 con 0x7f4004108680 2024-04-25T01:17:44.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4000003c20 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4004106630 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f400001a650 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.856+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4004106b70 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.857+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4000024ad0 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.857+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4004063890 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.857+0000 7f3ffaffd700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f3ff0039100 0x7f3ff003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.857+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f400004d5c0 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.857+0000 7f4009532700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f3ff0039100 0x7f3ff003b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.858+0000 7f4009532700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f3ff0039100 0x7f3ff003b5c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4006fd0 tx=0x7f3ff4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:38.862+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f400000f2e0 con 0x7f4004108680 2024-04-25T01:17:44.165 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:39.074+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) -- 0x7f40041073d0 con 0x7f3ff0039100 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.565+0000 7f3ffaffd700 1 -- 172.21.15.16:0/988892930 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f40041073d0 con 0x7f3ff0039100 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f3ff0039100 msgr2=0x7f3ff003b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f3ff0039100 0x7f3ff003b5c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4006fd0 tx=0x7f3ff4006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 msgr2=0x7f4004107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f4004107f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4000011a50 tx=0x7f4000011b30 comp rx=0 tx=0).stop 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 shutdown_connections 2024-04-25T01:17:44.166 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f3ff0039100 0x7f3ff003b5c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.167 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 --2- 172.21.15.16:0/988892930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4004108680 0x7f4004107f00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.167 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 >> 172.21.15.16:0/988892930 conn(0x7f4004102050 msgr2=0x7f4004102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:44.167 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 shutdown_connections 2024-04-25T01:17:44.167 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:43.567+0000 7f400bf97700 1 -- 172.21.15.16:0/988892930 wait complete. 2024-04-25T01:17:44.167 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-04-25T01:17:44.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:44 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:44.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:44 smithi016 ceph-mon[29153]: Added host smithi016 2024-04-25T01:17:44.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:44 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:46.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:45 smithi016 ceph-mon[29153]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:46.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:45 smithi016 ceph-mon[29153]: Saving service mon spec with placement count:5 2024-04-25T01:17:46.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:45 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:46.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T01:17:46.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facb1b87700 1 Processor -- start 2024-04-25T01:17:46.426 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facb1b87700 1 -- start start 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facb1b87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facb1b87700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facac07c8b0 con 0x7facac07de80 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39496/0 (socket says 172.21.15.16:39496) 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.695+0000 7facab7fe700 1 -- 172.21.15.16:0/2546361055 learned_addr learned my addr 172.21.15.16:0/2546361055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facab7fe700 1 -- 172.21.15.16:0/2546361055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac07ca30 con 0x7facac07de80 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facab7fe700 1 --2- 172.21.15.16:0/2546361055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac07c2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0098a0 tx=0x7fac9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c82499369578de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facaa7fc700 1 -- 172.21.15.16:0/2546361055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac9c004160 con 0x7facac07de80 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facaa7fc700 1 -- 172.21.15.16:0/2546361055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fac9c004300 con 0x7facac07de80 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facaa7fc700 1 -- 172.21.15.16:0/2546361055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac9c011a80 con 0x7facac07de80 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facb1b87700 1 -- 172.21.15.16:0/2546361055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 msgr2=0x7facac07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.427 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facb1b87700 1 --2- 172.21.15.16:0/2546361055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac07c2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0098a0 tx=0x7fac9c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facb1b87700 1 -- 172.21.15.16:0/2546361055 shutdown_connections 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facb1b87700 1 --2- 172.21.15.16:0/2546361055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac07c2e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.696+0000 7facb1b87700 1 -- 172.21.15.16:0/2546361055 >> 172.21.15.16:0/2546361055 conn(0x7facac103180 msgr2=0x7facac1055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.697+0000 7facb1b87700 1 -- 172.21.15.16:0/2546361055 shutdown_connections 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.697+0000 7facb1b87700 1 -- 172.21.15.16:0/2546361055 wait complete. 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.697+0000 7facb1b87700 1 Processor -- start 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.697+0000 7facb1b87700 1 -- start start 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.697+0000 7facb1b87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.697+0000 7facb1b87700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facac10e160 con 0x7facac07de80 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac19dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39502/0 (socket says 172.21.15.16:39502) 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facab7fe700 1 -- 172.21.15.16:0/2359918046 learned_addr learned my addr 172.21.15.16:0/2359918046 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facab7fe700 1 -- 172.21.15.16:0/2359918046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac19e230 con 0x7facac07de80 2024-04-25T01:17:46.428 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facab7fe700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac19dcf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac9c0039f0 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fac9c003b90 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac9c01a740 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facac19e4c0 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.698+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facac1a11a0 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.699+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fac9c01a9f0 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.699+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facac063890 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.699+0000 7faca8ff9700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fac980390a0 0x7fac9803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.699+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fac9c04d4c0 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.699+0000 7facaaffd700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fac980390a0 0x7fac9803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.700+0000 7facaaffd700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fac980390a0 0x7fac9803b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fac94006fd0 tx=0x7fac94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.704+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fac9c01ace0 con 0x7facac07de80 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.932+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7facac104380 con 0x7fac980390a0 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.938+0000 7faca8ff9700 1 -- 172.21.15.16:0/2359918046 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7facac104380 con 0x7fac980390a0 2024-04-25T01:17:46.429 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fac980390a0 msgr2=0x7fac9803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fac980390a0 0x7fac9803b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fac94006fd0 tx=0x7fac94006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 msgr2=0x7facac19dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac19dcf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c011b00 comp rx=0 tx=0).stop 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 shutdown_connections 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fac980390a0 0x7fac9803b560 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 --2- 172.21.15.16:0/2359918046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facac07de80 0x7facac19dcf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 >> 172.21.15.16:0/2359918046 conn(0x7facac103180 msgr2=0x7facac103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 shutdown_connections 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:44.940+0000 7facb1b87700 1 -- 172.21.15.16:0/2359918046 wait complete. 2024-04-25T01:17:46.430 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-04-25T01:17:47.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:47 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:47.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:47 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:47.716 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.891+0000 7fc0e7d2a700 1 Processor -- start 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.891+0000 7fc0e7d2a700 1 -- start start 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e7d2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e0106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e7d2a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0107490 con 0x7fc0e0106aa0 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e5ac6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e0106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e5ac6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e0106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39508/0 (socket says 172.21.15.16:39508) 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e5ac6700 1 -- 172.21.15.16:0/3768919382 learned_addr learned my addr 172.21.15.16:0/3768919382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e5ac6700 1 -- 172.21.15.16:0/3768919382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e0107ca0 con 0x7fc0e0106aa0 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e5ac6700 1 --2- 172.21.15.16:0/3768919382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e0106ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40098a0 tx=0x7fc0d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef4099db378be34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e4ac4700 1 -- 172.21.15.16:0/3768919382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0d4004160 con 0x7fc0e0106aa0 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e4ac4700 1 -- 172.21.15.16:0/3768919382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0d4004300 con 0x7fc0e0106aa0 2024-04-25T01:17:47.717 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.892+0000 7fc0e4ac4700 1 -- 172.21.15.16:0/3768919382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0d4011a50 con 0x7fc0e0106aa0 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3768919382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 msgr2=0x7fc0e0106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 --2- 172.21.15.16:0/3768919382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e0106ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40098a0 tx=0x7fc0d4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3768919382 shutdown_connections 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 --2- 172.21.15.16:0/3768919382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e0106ec0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3768919382 >> 172.21.15.16:0/3768919382 conn(0x7fc0e0102060 msgr2=0x7fc0e0104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3768919382 shutdown_connections 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.893+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3768919382 wait complete. 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e7d2a700 1 Processor -- start 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e7d2a700 1 -- start start 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e7d2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e007ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e7d2a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e010b050 con 0x7fc0e0106aa0 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e5ac6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e007ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.718 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e5ac6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e007ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39524/0 (socket says 172.21.15.16:39524) 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e5ac6700 1 -- 172.21.15.16:0/3498542249 learned_addr learned my addr 172.21.15.16:0/3498542249 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.894+0000 7fc0e5ac6700 1 -- 172.21.15.16:0/3498542249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e019e790 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.895+0000 7fc0e5ac6700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e007ba50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006ba0 tx=0x7fc0d4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.895+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0d4003af0 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.895+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e019e960 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.895+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0d401a8e0 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.895+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0d4024900 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.895+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e019ed80 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.896+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc0d4022020 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.896+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e0063890 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.896+0000 7fc0d2ffd700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fc0cc039000 0x7fc0cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.896+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc0d404d5f0 con 0x7fc0e0106aa0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.896+0000 7fc0e52c5700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fc0cc039000 0x7fc0cc03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.719 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.897+0000 7fc0e52c5700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fc0cc039000 0x7fc0cc03b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc006fd0 tx=0x7fc0dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:46.901+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc0d400b5a0 con 0x7fc0e0106aa0 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.124+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc0e01031f0 con 0x7fc0cc039000 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.129+0000 7fc0d2ffd700 1 -- 172.21.15.16:0/3498542249 <== mgr.14120 v2:172.21.15.16:6800/1190188268 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc0e01031f0 con 0x7fc0cc039000 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fc0cc039000 msgr2=0x7fc0cc03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fc0cc039000 0x7fc0cc03b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc006fd0 tx=0x7fc0dc006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 msgr2=0x7fc0e007ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e007ba50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006ba0 tx=0x7fc0d4011ad0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 shutdown_connections 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fc0cc039000 0x7fc0cc03b4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 --2- 172.21.15.16:0/3498542249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e0106aa0 0x7fc0e007ba50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 >> 172.21.15.16:0/3498542249 conn(0x7fc0e0102060 msgr2=0x7fc0e0102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 shutdown_connections 2024-04-25T01:17:47.720 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:47.132+0000 7fc0e7d2a700 1 -- 172.21.15.16:0/3498542249 wait complete. 2024-04-25T01:17:48.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:48 smithi016 ceph-mon[29153]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:17:48.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:48 smithi016 ceph-mon[29153]: Saving service mgr spec with placement count:2 2024-04-25T01:17:48.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:48 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:49.510 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:49 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3175250080' entity='client.admin' 2024-04-25T01:17:49.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:49 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:49.518 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.163+0000 7fbdacfb6700 1 Processor -- start 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.163+0000 7fbdacfb6700 1 -- start start 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbdacfb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda8107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbdacfb6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8108590 con 0x7fbda8107ba0 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbda659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda8107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbda659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda8107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39536/0 (socket says 172.21.15.16:39536) 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbda659c700 1 -- 172.21.15.16:0/3788619861 learned_addr learned my addr 172.21.15.16:0/3788619861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbda659c700 1 -- 172.21.15.16:0/3788619861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda8108da0 con 0x7fbda8107ba0 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbda659c700 1 --2- 172.21.15.16:0/3788619861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda8107fc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd980098a0 tx=0x7fbd98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e555a7671f7ade server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.164+0000 7fbda559a700 1 -- 172.21.15.16:0/3788619861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd98004160 con 0x7fbda8107ba0 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbda559a700 1 -- 172.21.15.16:0/3788619861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd98004300 con 0x7fbda8107ba0 2024-04-25T01:17:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbda559a700 1 -- 172.21.15.16:0/3788619861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd98011a50 con 0x7fbda8107ba0 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3788619861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 msgr2=0x7fbda8107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 --2- 172.21.15.16:0/3788619861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda8107fc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd980098a0 tx=0x7fbd98009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3788619861 shutdown_connections 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 --2- 172.21.15.16:0/3788619861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda8107fc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3788619861 >> 172.21.15.16:0/3788619861 conn(0x7fbda8103180 msgr2=0x7fbda81055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3788619861 shutdown_connections 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.165+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3788619861 wait complete. 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbdacfb6700 1 Processor -- start 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbdacfb6700 1 -- start start 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbdacfb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbdacfb6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbda810c150 con 0x7fbda8107ba0 2024-04-25T01:17:49.520 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbda659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbda659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda819ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39546/0 (socket says 172.21.15.16:39546) 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.166+0000 7fbda659c700 1 -- 172.21.15.16:0/3175250080 learned_addr learned my addr 172.21.15.16:0/3175250080 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbda659c700 1 -- 172.21.15.16:0/3175250080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda819bfb0 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbda659c700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda819ba70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbd98006ba0 tx=0x7fbd98011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd98003af0 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd9801a8e0 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda819c240 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd98024900 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.167+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda819ef30 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.168+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fbd98022020 con 0x7fbda8107ba0 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.168+0000 7fbd977fe700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fbd90039000 0x7fbd9003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.521 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.168+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda8063890 con 0x7fbda8107ba0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.168+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbd9804d420 con 0x7fbda8107ba0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.168+0000 7fbda5d9b700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fbd90039000 0x7fbd9003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.169+0000 7fbda5d9b700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fbd90039000 0x7fbd9003b4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006fd0 tx=0x7fbd9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.173+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd9800f2e0 con 0x7fbda8107ba0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.402+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fbda819f4d0 con 0x7fbda8107ba0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.404+0000 7fbd977fe700 1 -- 172.21.15.16:0/3175250080 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fbd9802d3f0 con 0x7fbda8107ba0 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fbd90039000 msgr2=0x7fbd9003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fbd90039000 0x7fbd9003b4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006fd0 tx=0x7fbd9c006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 msgr2=0x7fbda819ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda819ba70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbd98006ba0 tx=0x7fbd98011ad0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 shutdown_connections 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fbd90039000 0x7fbd9003b4c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.522 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 --2- 172.21.15.16:0/3175250080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda8107ba0 0x7fbda819ba70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 >> 172.21.15.16:0/3175250080 conn(0x7fbda8103180 msgr2=0x7fbda8103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.406+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 shutdown_connections 2024-04-25T01:17:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:48.407+0000 7fbdacfb6700 1 -- 172.21.15.16:0/3175250080 wait complete. 2024-04-25T01:17:50.824 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4f6fc700 1 Processor -- start 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4f6fc700 1 -- start start 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4f6fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab4810aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4f6fc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab4807bdf0 con 0x7fab48108630 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4d498700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab4810aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4d498700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab4810aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39556/0 (socket says 172.21.15.16:39556) 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.012+0000 7fab4d498700 1 -- 172.21.15.16:0/1789531917 learned_addr learned my addr 172.21.15.16:0/1789531917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.013+0000 7fab4d498700 1 -- 172.21.15.16:0/1789531917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4810af60 con 0x7fab48108630 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.013+0000 7fab4d498700 1 --2- 172.21.15.16:0/1789531917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab4810aa20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab380098a0 tx=0x7fab38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60fc71ed37785f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.013+0000 7fab3ffff700 1 -- 172.21.15.16:0/1789531917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab38004160 con 0x7fab48108630 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.013+0000 7fab3ffff700 1 -- 172.21.15.16:0/1789531917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab38004300 con 0x7fab48108630 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.013+0000 7fab3ffff700 1 -- 172.21.15.16:0/1789531917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab38011a50 con 0x7fab48108630 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 -- 172.21.15.16:0/1789531917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 msgr2=0x7fab4810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 --2- 172.21.15.16:0/1789531917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab4810aa20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab380098a0 tx=0x7fab38009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 -- 172.21.15.16:0/1789531917 shutdown_connections 2024-04-25T01:17:50.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 --2- 172.21.15.16:0/1789531917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab4810aa20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 -- 172.21.15.16:0/1789531917 >> 172.21.15.16:0/1789531917 conn(0x7fab48102040 msgr2=0x7fab48104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 -- 172.21.15.16:0/1789531917 shutdown_connections 2024-04-25T01:17:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.014+0000 7fab4f6fc700 1 -- 172.21.15.16:0/1789531917 wait complete. 2024-04-25T01:17:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4f6fc700 1 Processor -- start 2024-04-25T01:17:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4f6fc700 1 -- start start 2024-04-25T01:17:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4f6fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab48107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.898 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4f6fc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab4810bf10 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4d498700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab48107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4d498700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab48107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39558/0 (socket says 172.21.15.16:39558) 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4d498700 1 -- 172.21.15.16:0/2458022566 learned_addr learned my addr 172.21.15.16:0/2458022566 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4d498700 1 -- 172.21.15.16:0/2458022566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab481083f0 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.015+0000 7fab4d498700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab48107eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fab38011a20 tx=0x7fab38011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.016+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab38003b70 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.016+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab3801a8e0 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.016+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab481065e0 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.016+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab38024990 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.016+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab48106b20 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.016+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fab38022020 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.017+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab48063890 con 0x7fab48108630 2024-04-25T01:17:50.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.017+0000 7fab3e7fc700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fab34039050 0x7fab3403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.017+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fab3804d740 con 0x7fab48108630 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.017+0000 7fab4cc97700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fab34039050 0x7fab3403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.017+0000 7fab4cc97700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fab34039050 0x7fab3403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab44006fd0 tx=0x7fab44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.022+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fab38020c10 con 0x7fab48108630 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.242+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fab48106e60 con 0x7fab48108630 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.245+0000 7fab3e7fc700 1 -- 172.21.15.16:0/2458022566 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fab48106e60 con 0x7fab48108630 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fab34039050 msgr2=0x7fab3403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fab34039050 0x7fab3403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab44006fd0 tx=0x7fab44006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 msgr2=0x7fab48107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab48107eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fab38011a20 tx=0x7fab38011b00 comp rx=0 tx=0).stop 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 shutdown_connections 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7fab34039050 0x7fab3403b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 --2- 172.21.15.16:0/2458022566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab48108630 0x7fab48107eb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 >> 172.21.15.16:0/2458022566 conn(0x7fab48102040 msgr2=0x7fab48102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:50.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 shutdown_connections 2024-04-25T01:17:50.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:50.248+0000 7fab4f6fc700 1 -- 172.21.15.16:0/2458022566 wait complete. 2024-04-25T01:17:50.901 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-04-25T01:17:51.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:51 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2458022566' entity='client.admin' 2024-04-25T01:17:51.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:51 smithi016 ceph-mon[29153]: from='mgr.14120 172.21.15.16:0/187786147' entity='mgr.x' 2024-04-25T01:17:52.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-mon[29153]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:17:52.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/423828440' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T01:17:52.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: ignoring --setuser ceph since I am not root 2024-04-25T01:17:52.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: ignoring --setgroup ceph since I am not root 2024-04-25T01:17:52.743 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:52.593+0000 7f039b34d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:17:52.997 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:52.785+0000 7f039b34d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:52.997 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:52.890+0000 7f039b34d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.303+0000 7f075440c700 1 Processor -- start 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.303+0000 7f075440c700 1 -- start start 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f075440c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f075440c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f074c108550 con 0x7f074c107b60 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f07521a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f07521a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39574/0 (socket says 172.21.15.16:39574) 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f07521a8700 1 -- 172.21.15.16:0/3983609629 learned_addr learned my addr 172.21.15.16:0/3983609629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f07521a8700 1 -- 172.21.15.16:0/3983609629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074c108d60 con 0x7f074c107b60 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.304+0000 7f07521a8700 1 --2- 172.21.15.16:0/3983609629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c107f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07480098a0 tx=0x7f0748009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414de47a3c6fc2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.305+0000 7f07511a6700 1 -- 172.21.15.16:0/3983609629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0748004160 con 0x7f074c107b60 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.305+0000 7f07511a6700 1 -- 172.21.15.16:0/3983609629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0748004300 con 0x7f074c107b60 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.305+0000 7f07511a6700 1 -- 172.21.15.16:0/3983609629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0748011a50 con 0x7f074c107b60 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 -- 172.21.15.16:0/3983609629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 msgr2=0x7f074c107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 --2- 172.21.15.16:0/3983609629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c107f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07480098a0 tx=0x7f0748009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 -- 172.21.15.16:0/3983609629 shutdown_connections 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 --2- 172.21.15.16:0/3983609629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c107f80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 -- 172.21.15.16:0/3983609629 >> 172.21.15.16:0/3983609629 conn(0x7f074c103160 msgr2=0x7f074c105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 -- 172.21.15.16:0/3983609629 shutdown_connections 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.306+0000 7f075440c700 1 -- 172.21.15.16:0/3983609629 wait complete. 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f075440c700 1 Processor -- start 2024-04-25T01:17:53.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f075440c700 1 -- start start 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f075440c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c19ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f075440c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c110 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f07521a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c19ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f07521a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c19ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39584/0 (socket says 172.21.15.16:39584) 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f07521a8700 1 -- 172.21.15.16:0/423828440 learned_addr learned my addr 172.21.15.16:0/423828440 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.307+0000 7f07521a8700 1 -- 172.21.15.16:0/423828440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074c19bfd0 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.308+0000 7f07521a8700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c19ba90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0748006ba0 tx=0x7f0748011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.308+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0748003af0 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.308+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f074801a8e0 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.308+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0748024900 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.308+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074c19c260 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.308+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f074c19ef50 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.309+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0748022020 con 0x7f074c107b60 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.309+0000 7f07437fe700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f0738039000 0x7f073803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.309+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f074804e360 con 0x7f074c107b60 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.309+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f074c063890 con 0x7f074c107b60 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.309+0000 7f07519a7700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f0738039000 0x7f073803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.310+0000 7f07519a7700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f0738039000 0x7f073803b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f073c006fd0 tx=0x7f073c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.315+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0748020c10 con 0x7f074c107b60 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:51.586+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f074c19f450 con 0x7f074c107b60 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.403+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f074800b5a0 con 0x7f074c107b60 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.405+0000 7f07437fe700 1 -- 172.21.15.16:0/423828440 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f074802dc90 con 0x7f074c107b60 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f0738039000 msgr2=0x7f073803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f0738039000 0x7f073803b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f073c006fd0 tx=0x7f073c006e40 comp rx=0 tx=0).stop 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 msgr2=0x7f074c19ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c19ba90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0748006ba0 tx=0x7f0748011ad0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 shutdown_connections 2024-04-25T01:17:53.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f0738039000 0x7f073803b4c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 --2- 172.21.15.16:0/423828440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f074c107b60 0x7f074c19ba90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 >> 172.21.15.16:0/423828440 conn(0x7f074c103160 msgr2=0x7f074c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:53.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 shutdown_connections 2024-04-25T01:17:53.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:52.407+0000 7f075440c700 1 -- 172.21.15.16:0/423828440 wait complete. 2024-04-25T01:17:53.247 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:53.215+0000 7f039b34d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:17:53.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:53.306+0000 7f039b34d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:17:53.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:53 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/423828440' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T01:17:53.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:53 smithi016 ceph-mon[29153]: mgrmap e9: x(active, since 21s) 2024-04-25T01:17:54.349 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.471+0000 7f28cee78700 1 Processor -- start 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.471+0000 7f28cee78700 1 -- start start 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.471+0000 7f28cee78700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c8107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.471+0000 7f28cee78700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28c8108560 con 0x7f28c8107b70 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.471+0000 7f28ccc14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c8107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28ccc14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c8107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57492/0 (socket says 172.21.15.16:57492) 2024-04-25T01:17:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28ccc14700 1 -- 172.21.15.16:0/3616629655 learned_addr learned my addr 172.21.15.16:0/3616629655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28ccc14700 1 -- 172.21.15.16:0/3616629655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c8108d70 con 0x7f28c8107b70 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28ccc14700 1 --2- 172.21.15.16:0/3616629655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c8107f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0098a0 tx=0x7f28bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a3cfdbbebac431 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28c77fe700 1 -- 172.21.15.16:0/3616629655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28bc004160 con 0x7f28c8107b70 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28c77fe700 1 -- 172.21.15.16:0/3616629655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28bc004300 con 0x7f28c8107b70 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.472+0000 7f28c77fe700 1 -- 172.21.15.16:0/3616629655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28bc011a50 con 0x7f28c8107b70 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 -- 172.21.15.16:0/3616629655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 msgr2=0x7f28c8107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 --2- 172.21.15.16:0/3616629655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c8107f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0098a0 tx=0x7f28bc009bb0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 -- 172.21.15.16:0/3616629655 shutdown_connections 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 --2- 172.21.15.16:0/3616629655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c8107f90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 -- 172.21.15.16:0/3616629655 >> 172.21.15.16:0/3616629655 conn(0x7f28c8103130 msgr2=0x7f28c8105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 -- 172.21.15.16:0/3616629655 shutdown_connections 2024-04-25T01:17:54.351 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.473+0000 7f28cee78700 1 -- 172.21.15.16:0/3616629655 wait complete. 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28cee78700 1 Processor -- start 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28cee78700 1 -- start start 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28cee78700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28cee78700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28c810c120 con 0x7f28c8107b70 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28ccc14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28ccc14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c819dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57498/0 (socket says 172.21.15.16:57498) 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28ccc14700 1 -- 172.21.15.16:0/3611088865 learned_addr learned my addr 172.21.15.16:0/3611088865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.474+0000 7f28ccc14700 1 -- 172.21.15.16:0/3611088865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c819e230 con 0x7f28c8107b70 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.475+0000 7f28ccc14700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c819dcf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f28bc011710 tx=0x7f28bc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.475+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28bc0039c0 con 0x7f28c8107b70 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.475+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c819e4c0 con 0x7f28c8107b70 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.475+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28bc003b60 con 0x7f28c8107b70 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.475+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28bc01a690 con 0x7f28c8107b70 2024-04-25T01:17:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.475+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c81a11a0 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.476+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c8063890 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.476+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f28bc01a990 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.476+0000 7f28c5ffb700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 0x7f28b803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.476+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f28bc04d720 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.476+0000 7f28c7fff700 1 -- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 msgr2=0x7f28b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.476+0000 7f28c7fff700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 0x7f28b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.484+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f28bc019c70 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.677+0000 7f28c7fff700 1 -- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 msgr2=0x7f28b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.677+0000 7f28c7fff700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 0x7f28b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.796+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f28c81a1750 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.797+0000 7f28c5ffb700 1 -- 172.21.15.16:0/3611088865 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f28bc016df0 con 0x7f28c8107b70 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 msgr2=0x7f28b803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:17:54.353 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 0x7f28b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 msgr2=0x7f28c819dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c819dcf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f28bc011710 tx=0x7f28bc011ad0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 shutdown_connections 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f28b8039050 0x7f28b803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 --2- 172.21.15.16:0/3611088865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28c8107b70 0x7f28c819dcf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 >> 172.21.15.16:0/3611088865 conn(0x7f28c8103130 msgr2=0x7f28c8103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 shutdown_connections 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:53.799+0000 7f28cee78700 1 -- 172.21.15.16:0/3611088865 wait complete. 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-25T01:17:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-04-25T01:17:54.464 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:54 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3611088865' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:17:54.738 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:54.573+0000 7f039b34d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:17:54.738 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:54.754+0000 7f039b34d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:55.010 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:54.933+0000 7f039b34d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:17:55.313 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:55.234+0000 7f039b34d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:17:55.313 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:55.330+0000 7f039b34d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:17:55.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:55.540+0000 7f039b34d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:17:56.119 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:55.740+0000 7f039b34d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:17:56.614 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:56.364+0000 7f039b34d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:17:57.098 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:56.806+0000 7f039b34d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:17:57.098 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:56.906+0000 7f039b34d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:17:57.098 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:57 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:57.114+0000 7f039b34d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:17:57.990 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:57 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:57.657+0000 7f039b34d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:17:57.990 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:57 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:57.753+0000 7f039b34d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:17:58.346 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:58.003+0000 7f039b34d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:17:58.346 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:58.142+0000 7f039b34d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:17:58.346 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:58.241+0000 7f039b34d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:17:58.346 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:58.361+0000 7f039b34d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:17:58.613 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:58.457+0000 7f039b34d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:17:59.561 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:59.299+0000 7f039b34d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:17:59.561 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:17:59.523+0000 7f039b34d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:17:59.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: Active manager daemon x restarted 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: Activating manager daemon x 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: mgrmap e10: x(active, starting, since 0.00360269s) 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: Manager daemon x is now available 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:17:59.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:17:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:18:00.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:18:00.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:00 smithi016 ceph-mon[29153]: [25/Apr/2024:01:18:00] ENGINE Bus STARTING 2024-04-25T01:18:00.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:00 smithi016 ceph-mon[29153]: [25/Apr/2024:01:18:00] ENGINE Serving on https://172.21.15.16:7150 2024-04-25T01:18:00.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:00 smithi016 ceph-mon[29153]: mgrmap e11: x(active, since 1.00643s) 2024-04-25T01:18:01.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:18:01.458 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72f491c700 1 Processor -- start 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72f491c700 1 -- start start 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72f491c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f0108430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72f491c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f72f0108a00 con 0x7f72f0108010 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72ee59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f0108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72ee59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f0108430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57500/0 (socket says 172.21.15.16:57500) 2024-04-25T01:18:01.459 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.821+0000 7f72ee59c700 1 -- 172.21.15.16:0/1119033498 learned_addr learned my addr 172.21.15.16:0/1119033498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.822+0000 7f72ee59c700 1 -- 172.21.15.16:0/1119033498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f0108b80 con 0x7f72f0108010 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.822+0000 7f72ee59c700 1 --2- 172.21.15.16:0/1119033498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f0108430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72e00098a0 tx=0x7f72e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff15d827ea2a7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.822+0000 7f72ed59a700 1 -- 172.21.15.16:0/1119033498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72e0004160 con 0x7f72f0108010 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.822+0000 7f72ed59a700 1 -- 172.21.15.16:0/1119033498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72e0004300 con 0x7f72f0108010 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.822+0000 7f72ed59a700 1 -- 172.21.15.16:0/1119033498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72e0011a50 con 0x7f72f0108010 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 -- 172.21.15.16:0/1119033498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 msgr2=0x7f72f0108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 --2- 172.21.15.16:0/1119033498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f0108430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72e00098a0 tx=0x7f72e0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 -- 172.21.15.16:0/1119033498 shutdown_connections 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 --2- 172.21.15.16:0/1119033498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f0108430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 -- 172.21.15.16:0/1119033498 >> 172.21.15.16:0/1119033498 conn(0x7f72f0101780 msgr2=0x7f72f0103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 -- 172.21.15.16:0/1119033498 shutdown_connections 2024-04-25T01:18:01.460 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.823+0000 7f72f491c700 1 -- 172.21.15.16:0/1119033498 wait complete. 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72f491c700 1 Processor -- start 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72f491c700 1 -- start start 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72f491c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72f491c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01069d0 con 0x7f72f0108010 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72ee59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72ee59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f019db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57506/0 (socket says 172.21.15.16:57506) 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72ee59c700 1 -- 172.21.15.16:0/2371966067 learned_addr learned my addr 172.21.15.16:0/2371966067 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.824+0000 7f72ee59c700 1 -- 172.21.15.16:0/2371966067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f019e090 con 0x7f72f0108010 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.825+0000 7f72ee59c700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f019db50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f72e0011a20 tx=0x7f72e0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:01.461 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.825+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72e0003b70 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.825+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72e001a8e0 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.825+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f019e380 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.825+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72e0024990 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.825+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f01a1010 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.826+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f72e0022020 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.826+0000 7f72df7fe700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.826+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 --> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f72d803bc20 con 0x7f72d8039050 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.826+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f72e004d8b0 con 0x7f72f0108010 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.826+0000 7f72edd9b700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 msgr2=0x7f72d803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:18:01.462 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:54.826+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:55.027+0000 7f72edd9b700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 msgr2=0x7f72d803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:55.027+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:55.428+0000 7f72edd9b700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 msgr2=0x7f72d803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:55.428+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:56.229+0000 7f72edd9b700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 msgr2=0x7f72d803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:56.229+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:57.830+0000 7f72edd9b700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 msgr2=0x7f72d803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1190188268 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:57.830+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:59.527+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f72e00193c0 con 0x7f72f0108010 2024-04-25T01:18:01.463 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:59.527+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 msgr2=0x7f72d803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:17:59.527+0000 7f72df7fe700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.528+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f72e004f700 con 0x7f72f0108010 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.529+0000 7f72df7fe700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f72d803cb10 0x7f72d803ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.529+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f72d803bc20 con 0x7f72d803cb10 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.529+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f72d803cb10 0x7f72d803ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.530+0000 7f72edd9b700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f72d803cb10 0x7f72d803ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72e4003a10 tx=0x7f72e40092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.530+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f72d803bc20 con 0x7f72d803cb10 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.537+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f72f0063890 con 0x7f72d803cb10 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.537+0000 7f72df7fe700 1 -- 172.21.15.16:0/2371966067 <== mgr.14152 v2:172.21.15.16:6800/1051975299 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f72f0063890 con 0x7f72d803cb10 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f72d803cb10 msgr2=0x7f72d803ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f72d803cb10 0x7f72d803ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72e4003a10 tx=0x7f72e40092b0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.464 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 msgr2=0x7f72f019db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f019db50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f72e0011a20 tx=0x7f72e0011b00 comp rx=0 tx=0).stop 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 shutdown_connections 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72f0108010 0x7f72f019db50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f72d803cb10 0x7f72d803ef00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 --2- 172.21.15.16:0/2371966067 >> [v2:172.21.15.16:6800/1190188268,v1:172.21.15.16:6801/1190188268] conn(0x7f72d8039050 0x7f72d803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 >> 172.21.15.16:0/2371966067 conn(0x7f72f0101780 msgr2=0x7f72f0194190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 shutdown_connections 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:00.538+0000 7f72f491c700 1 -- 172.21.15.16:0/2371966067 wait complete. 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-04-25T01:18:01.465 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-04-25T01:18:02.300 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:02 smithi016 ceph-mon[29153]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:18:02.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:02 smithi016 ceph-mon[29153]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:18:02.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:02 smithi016 ceph-mon[29153]: [25/Apr/2024:01:18:00] ENGINE Serving on http://172.21.15.16:8765 2024-04-25T01:18:02.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:02 smithi016 ceph-mon[29153]: [25/Apr/2024:01:18:00] ENGINE Bus STARTED 2024-04-25T01:18:02.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:02.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T01:18:02.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0f9e9700 1 Processor -- start 2024-04-25T01:18:02.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0f9e9700 1 -- start start 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0f9e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0f9e9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0807bdf0 con 0x7f7a08108650 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0d785700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0d785700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41528/0 (socket says 172.21.15.16:41528) 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.942+0000 7f7a0d785700 1 -- 172.21.15.16:0/57625546 learned_addr learned my addr 172.21.15.16:0/57625546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.943+0000 7f7a0d785700 1 -- 172.21.15.16:0/57625546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0810af80 con 0x7f7a08108650 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.943+0000 7f7a0d785700 1 --2- 172.21.15.16:0/57625546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0810aa40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a040098a0 tx=0x7f7a04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c186b573e2330b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.943+0000 7f79fffff700 1 -- 172.21.15.16:0/57625546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a04004160 con 0x7f7a08108650 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.943+0000 7f79fffff700 1 -- 172.21.15.16:0/57625546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a04004300 con 0x7f7a08108650 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.943+0000 7f79fffff700 1 -- 172.21.15.16:0/57625546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a04011a50 con 0x7f7a08108650 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/57625546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 msgr2=0x7f7a0810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 --2- 172.21.15.16:0/57625546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0810aa40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a040098a0 tx=0x7f7a04009bb0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/57625546 shutdown_connections 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 --2- 172.21.15.16:0/57625546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0810aa40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/57625546 >> 172.21.15.16:0/57625546 conn(0x7f7a08102020 msgr2=0x7f7a08104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/57625546 shutdown_connections 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.944+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/57625546 wait complete. 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0f9e9700 1 Processor -- start 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0f9e9700 1 -- start start 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0f9e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0f9e9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0810bf30 con 0x7f7a08108650 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0d785700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0d785700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0819ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41532/0 (socket says 172.21.15.16:41532) 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0d785700 1 -- 172.21.15.16:0/3593857267 learned_addr learned my addr 172.21.15.16:0/3593857267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0d785700 1 -- 172.21.15.16:0/3593857267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0819bf90 con 0x7f7a08108650 2024-04-25T01:18:02.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0d785700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0819ba50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a04006b20 tx=0x7f7a04011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a04003b70 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a0401a8e0 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a0819c220 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.945+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a040249b0 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.946+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a0819ef10 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.946+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f7a04022020 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.947+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a08063890 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.947+0000 7f79fe7fc700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f79f4038ff0 0x7f79f403b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.947+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7a0404d770 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.947+0000 7f7a0cf84700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f79f4038ff0 0x7f79f403b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.947+0000 7f7a0cf84700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f79f4038ff0 0x7f79f403b4b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79f8006fd0 tx=0x7f79f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:01.952+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a0400b3e0 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.137+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f7a0402da00 con 0x7f7a08108650 2024-04-25T01:18:02.975 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.166+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f7a08103180 con 0x7f79f4038ff0 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.317+0000 7f79fe7fc700 1 -- 172.21.15.16:0/3593857267 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f7a08103180 con 0x7f79f4038ff0 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f79f4038ff0 msgr2=0x7f79f403b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f79f4038ff0 0x7f79f403b4b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79f8006fd0 tx=0x7f79f8006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 msgr2=0x7f7a0819ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0819ba50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a04006b20 tx=0x7f7a04011ad0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 shutdown_connections 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f79f4038ff0 0x7f79f403b4b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 --2- 172.21.15.16:0/3593857267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a08108650 0x7f7a0819ba50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 >> 172.21.15.16:0/3593857267 conn(0x7f7a08102020 msgr2=0x7f7a08102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 shutdown_connections 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:02.319+0000 7f7a0f9e9700 1 -- 172.21.15.16:0/3593857267 wait complete. 2024-04-25T01:18:02.976 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-04-25T01:18:03.231 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:03 smithi016 ceph-mon[29153]: mgrmap e12: x(active, since 2s) 2024-04-25T01:18:03.232 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:03 smithi016 ceph-mon[29153]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:03.232 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:03.232 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:03.232 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:05.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:04 smithi016 ceph-mon[29153]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:05.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:05.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$yaNcXh2Xjl.cfaEeMKlzOOqJ484oaggX.HU8nhXkpOpuBn6z10RVe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714007883, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.457+0000 7f34a7c9d700 1 Processor -- start 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.457+0000 7f34a7c9d700 1 -- start start 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a7c9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0106e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a7c9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34a0107430 con 0x7f34a0106a40 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a5a39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a5a39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41542/0 (socket says 172.21.15.16:41542) 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a5a39700 1 -- 172.21.15.16:0/1759919331 learned_addr learned my addr 172.21.15.16:0/1759919331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a5a39700 1 -- 172.21.15.16:0/1759919331 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a0107c40 con 0x7f34a0106a40 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a5a39700 1 --2- 172.21.15.16:0/1759919331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0106e60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f349c0098a0 tx=0x7f349c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fdd3bf1cbda2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.458+0000 7f34a4a37700 1 -- 172.21.15.16:0/1759919331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f349c004160 con 0x7f34a0106a40 2024-04-25T01:18:05.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a4a37700 1 -- 172.21.15.16:0/1759919331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f349c004300 con 0x7f34a0106a40 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a4a37700 1 -- 172.21.15.16:0/1759919331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f349c011a50 con 0x7f34a0106a40 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 -- 172.21.15.16:0/1759919331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 msgr2=0x7f34a0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 --2- 172.21.15.16:0/1759919331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0106e60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f349c0098a0 tx=0x7f349c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 -- 172.21.15.16:0/1759919331 shutdown_connections 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 --2- 172.21.15.16:0/1759919331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0106e60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 -- 172.21.15.16:0/1759919331 >> 172.21.15.16:0/1759919331 conn(0x7f34a0101fc0 msgr2=0x7f34a0104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 -- 172.21.15.16:0/1759919331 shutdown_connections 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.459+0000 7f34a7c9d700 1 -- 172.21.15.16:0/1759919331 wait complete. 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a7c9d700 1 Processor -- start 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a7c9d700 1 -- start start 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a7c9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0191bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a7c9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34a010aff0 con 0x7f34a0106a40 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a5a39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0191bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a5a39700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0191bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41548/0 (socket says 172.21.15.16:41548) 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a5a39700 1 -- 172.21.15.16:0/4251176995 learned_addr learned my addr 172.21.15.16:0/4251176995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a5a39700 1 -- 172.21.15.16:0/4251176995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a01920f0 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.460+0000 7f34a5a39700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0191bb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f349c006ba0 tx=0x7f349c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.461+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f349c003af0 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.461+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f349c01a8e0 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.461+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a01902e0 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.461+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f349c024900 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.461+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a0190820 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.462+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f349c022020 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.462+0000 7f3492ffd700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f348c0390b0 0x7f348c03b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.462+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a0063890 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.462+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f349c04d530 con 0x7f34a0106a40 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.462+0000 7f34a5238700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f348c0390b0 0x7f348c03b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.462+0000 7f34a5238700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f348c0390b0 0x7f348c03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3494006fd0 tx=0x7f3494006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.467+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f349c00be50 con 0x7f34a0106a40 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.702+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f34a01958a0 con 0x7f348c0390b0 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.949+0000 7f3492ffd700 1 -- 172.21.15.16:0/4251176995 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f34a01958a0 con 0x7f348c0390b0 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.951+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f348c0390b0 msgr2=0x7f348c03b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.951+0000 7f34a7c9d700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f348c0390b0 0x7f348c03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3494006fd0 tx=0x7f3494006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.951+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 msgr2=0x7f34a0191bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.951+0000 7f34a7c9d700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0191bb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f349c006ba0 tx=0x7f349c011ad0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.952+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 shutdown_connections 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.952+0000 7f34a7c9d700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f348c0390b0 0x7f348c03b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.952+0000 7f34a7c9d700 1 --2- 172.21.15.16:0/4251176995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34a0106a40 0x7f34a0191bb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.952+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 >> 172.21.15.16:0/4251176995 conn(0x7f34a0101fc0 msgr2=0x7f34a0102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.952+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 shutdown_connections 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:03.952+0000 7f34a7c9d700 1 -- 172.21.15.16:0/4251176995 wait complete. 2024-04-25T01:18:05.196 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-04-25T01:18:06.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:05 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3173723795' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T01:18:06.433 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef34df700 1 Processor -- start 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef34df700 1 -- start start 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef34df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef34df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec107480 con 0x7fdeec106a90 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef127b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef127b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41550/0 (socket says 172.21.15.16:41550) 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.601+0000 7fdef127b700 1 -- 172.21.15.16:0/2145425378 learned_addr learned my addr 172.21.15.16:0/2145425378 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.602+0000 7fdef127b700 1 -- 172.21.15.16:0/2145425378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec107c90 con 0x7fdeec106a90 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.602+0000 7fdef127b700 1 --2- 172.21.15.16:0/2145425378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec106eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdee800bb40 tx=0x7fdee800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47d2c7f39f7e847 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.602+0000 7fdee3fff700 1 -- 172.21.15.16:0/2145425378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee800dbe0 con 0x7fdeec106a90 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.602+0000 7fdee3fff700 1 -- 172.21.15.16:0/2145425378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdee80044e0 con 0x7fdeec106a90 2024-04-25T01:18:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.602+0000 7fdee3fff700 1 -- 172.21.15.16:0/2145425378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee8010470 con 0x7fdeec106a90 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- 172.21.15.16:0/2145425378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 msgr2=0x7fdeec106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 --2- 172.21.15.16:0/2145425378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec106eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdee800bb40 tx=0x7fdee800be50 comp rx=0 tx=0).stop 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- 172.21.15.16:0/2145425378 shutdown_connections 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 --2- 172.21.15.16:0/2145425378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec106eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- 172.21.15.16:0/2145425378 >> 172.21.15.16:0/2145425378 conn(0x7fdeec102050 msgr2=0x7fdeec104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- 172.21.15.16:0/2145425378 shutdown_connections 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- 172.21.15.16:0/2145425378 wait complete. 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 Processor -- start 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- start start 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.603+0000 7fdef34df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec07af70 con 0x7fdeec106a90 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef127b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef127b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41552/0 (socket says 172.21.15.16:41552) 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef127b700 1 -- 172.21.15.16:0/3173723795 learned_addr learned my addr 172.21.15.16:0/3173723795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef127b700 1 -- 172.21.15.16:0/3173723795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec199db0 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef127b700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec199870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdee800d650 tx=0x7fdee8003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee8027070 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdee8010d50 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeec19a040 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdee8022bd0 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.604+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeec19a4a0 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.605+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeec063890 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.605+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fdee802a070 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.605+0000 7fdee27fc700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fded80390b0 0x7fded803b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.605+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdee801f070 con 0x7fdeec106a90 2024-04-25T01:18:06.436 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.606+0000 7fdef0a7a700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fded80390b0 0x7fded803b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.606+0000 7fdef0a7a700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fded80390b0 0x7fded803b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdedc006fd0 tx=0x7fdedc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.610+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdee8051e20 con 0x7fdeec106a90 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.816+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fdeec192dd0 con 0x7fdeec106a90 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.817+0000 7fdee27fc700 1 -- 172.21.15.16:0/3173723795 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fdeec192dd0 con 0x7fdeec106a90 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fded80390b0 msgr2=0x7fded803b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fded80390b0 0x7fded803b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdedc006fd0 tx=0x7fdedc006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 msgr2=0x7fdeec199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec199870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdee800d650 tx=0x7fdee8003fb0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 shutdown_connections 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fded80390b0 0x7fded803b570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 --2- 172.21.15.16:0/3173723795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeec106a90 0x7fdeec199870 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 >> 172.21.15.16:0/3173723795 conn(0x7fdeec102050 msgr2=0x7fdeec18feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:06.437 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 shutdown_connections 2024-04-25T01:18:06.438 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:05.819+0000 7fdef34df700 1 -- 172.21.15.16:0/3173723795 wait complete. 2024-04-25T01:18:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:18:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-25T01:18:06.452 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:18:06.452 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-25T01:18:06.452 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-25T01:18:06.452 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T01:18:06.454 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-04-25T01:18:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016:8443/ 2024-04-25T01:18:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-04-25T01:18:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: Password: lt2qm35sj6 2024-04-25T01:18:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:06.455 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config directory 2024-04-25T01:18:06.456 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-04-25T01:18:07.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:06 smithi016 ceph-mon[29153]: mgrmap e13: x(active, since 6s) 2024-04-25T01:18:07.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb089408700 1 Processor -- start 2024-04-25T01:18:07.672 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb089408700 1 -- start start 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb089408700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb084107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb089408700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb084108590 con 0x7fb084107ba0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb082ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb084107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb082ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb084107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41556/0 (socket says 172.21.15.16:41556) 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb082ffd700 1 -- 172.21.15.16:0/931803793 learned_addr learned my addr 172.21.15.16:0/931803793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.929+0000 7fb082ffd700 1 -- 172.21.15.16:0/931803793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb084108da0 con 0x7fb084107ba0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.930+0000 7fb082ffd700 1 --2- 172.21.15.16:0/931803793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb084107fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0740098a0 tx=0x7fb074009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e3a75e9a746b15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.930+0000 7fb081ffb700 1 -- 172.21.15.16:0/931803793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb074004160 con 0x7fb084107ba0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.930+0000 7fb081ffb700 1 -- 172.21.15.16:0/931803793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb074004300 con 0x7fb084107ba0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.930+0000 7fb081ffb700 1 -- 172.21.15.16:0/931803793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb074011a50 con 0x7fb084107ba0 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.930+0000 7fb089408700 1 -- 172.21.15.16:0/931803793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 msgr2=0x7fb084107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.930+0000 7fb089408700 1 --2- 172.21.15.16:0/931803793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb084107fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0740098a0 tx=0x7fb074009bb0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.673 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 -- 172.21.15.16:0/931803793 shutdown_connections 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 --2- 172.21.15.16:0/931803793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb084107fc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 -- 172.21.15.16:0/931803793 >> 172.21.15.16:0/931803793 conn(0x7fb084103180 msgr2=0x7fb0841055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 -- 172.21.15.16:0/931803793 shutdown_connections 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 -- 172.21.15.16:0/931803793 wait complete. 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 Processor -- start 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.931+0000 7fb089408700 1 -- start start 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb089408700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb08419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb089408700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb08410c150 con 0x7fb084107ba0 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb082ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb08419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb082ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb08419ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41560/0 (socket says 172.21.15.16:41560) 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb082ffd700 1 -- 172.21.15.16:0/975585587 learned_addr learned my addr 172.21.15.16:0/975585587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb082ffd700 1 -- 172.21.15.16:0/975585587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08419bfd0 con 0x7fb084107ba0 2024-04-25T01:18:07.674 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb082ffd700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb08419ba90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb074006ba0 tx=0x7fb074011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb074003af0 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb07401a8e0 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.932+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08419c260 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.933+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb074024900 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.933+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08419ef50 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.933+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb074022020 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.934+0000 7fb06bfff700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb06c039150 0x7fb06c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.934+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb07404d5f0 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.934+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb084063890 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.934+0000 7fb0827fc700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb06c039150 0x7fb06c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.934+0000 7fb0827fc700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb06c039150 0x7fb06c03b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb078006fd0 tx=0x7fb078006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:06.939+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb074020c10 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.148+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb08419f120 con 0x7fb084107ba0 2024-04-25T01:18:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.149+0000 7fb06bfff700 1 -- 172.21.15.16:0/975585587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb074021d80 con 0x7fb084107ba0 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb06c039150 msgr2=0x7fb06c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb06c039150 0x7fb06c03b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb078006fd0 tx=0x7fb078006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 msgr2=0x7fb08419ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb08419ba90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb074006ba0 tx=0x7fb074011ad0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 shutdown_connections 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb06c039150 0x7fb06c03b610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 --2- 172.21.15.16:0/975585587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb084107ba0 0x7fb08419ba90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 >> 172.21.15.16:0/975585587 conn(0x7fb084103180 msgr2=0x7fb084103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 shutdown_connections 2024-04-25T01:18:07.676 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:07.151+0000 7fb089408700 1 -- 172.21.15.16:0/975585587 wait complete. 2024-04-25T01:18:09.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe8fa7700 1 Processor -- start 2024-04-25T01:18:09.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe8fa7700 1 -- start start 2024-04-25T01:18:09.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe8fa7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe410aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe8fa7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe407be10 con 0x7fcbe4108640 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe410aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe410aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41570/0 (socket says 172.21.15.16:41570) 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.103+0000 7fcbe259c700 1 -- 172.21.15.16:0/3661368394 learned_addr learned my addr 172.21.15.16:0/3661368394 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.104+0000 7fcbe259c700 1 -- 172.21.15.16:0/3661368394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe410af70 con 0x7fcbe4108640 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.104+0000 7fcbe259c700 1 --2- 172.21.15.16:0/3661368394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe410aa30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40098a0 tx=0x7fcbd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd5b63fdc0cd220 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.104+0000 7fcbe159a700 1 -- 172.21.15.16:0/3661368394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd4004160 con 0x7fcbe4108640 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.104+0000 7fcbe159a700 1 -- 172.21.15.16:0/3661368394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbd4004300 con 0x7fcbe4108640 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.104+0000 7fcbe159a700 1 -- 172.21.15.16:0/3661368394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd4011a50 con 0x7fcbe4108640 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3661368394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 msgr2=0x7fcbe410aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 --2- 172.21.15.16:0/3661368394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe410aa30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40098a0 tx=0x7fcbd4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3661368394 shutdown_connections 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 --2- 172.21.15.16:0/3661368394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe410aa30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3661368394 >> 172.21.15.16:0/3661368394 conn(0x7fcbe4102050 msgr2=0x7fcbe4104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3661368394 shutdown_connections 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.105+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3661368394 wait complete. 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe8fa7700 1 Processor -- start 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe8fa7700 1 -- start start 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe8fa7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe8fa7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe410bf70 con 0x7fcbe4108640 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41576/0 (socket says 172.21.15.16:41576) 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe259c700 1 -- 172.21.15.16:0/3723401876 learned_addr learned my addr 172.21.15.16:0/3723401876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe259c700 1 -- 172.21.15.16:0/3723401876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe419bf70 con 0x7fcbe4108640 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.106+0000 7fcbe259c700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe419ba30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006b20 tx=0x7fcbd4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.107+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd4003b70 con 0x7fcbe4108640 2024-04-25T01:18:09.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.107+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe419c200 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.107+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbd401a8e0 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.107+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd40249b0 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.107+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe419eef0 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.108+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fcbd4022020 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.108+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe4063890 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.108+0000 7fcbd37fe700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcbcc039140 0x7fcbcc03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.108+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcbd404db10 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.108+0000 7fcbe1d9b700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcbcc039140 0x7fcbcc03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.108+0000 7fcbe1d9b700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcbcc039140 0x7fcbcc03b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8006fd0 tx=0x7fcbd8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.113+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcbd402d9b0 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.438+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fcbe419f0c0 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.440+0000 7fcbd37fe700 1 -- 172.21.15.16:0/3723401876 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fcbd402cc80 con 0x7fcbe4108640 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.442+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcbcc039140 msgr2=0x7fcbcc03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.442+0000 7fcbe8fa7700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcbcc039140 0x7fcbcc03b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8006fd0 tx=0x7fcbd8006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.442+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 msgr2=0x7fcbe419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.442+0000 7fcbe8fa7700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe419ba30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006b20 tx=0x7fcbd4011ad0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.443+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 shutdown_connections 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.443+0000 7fcbe8fa7700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcbcc039140 0x7fcbcc03b600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.443+0000 7fcbe8fa7700 1 --2- 172.21.15.16:0/3723401876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbe4108640 0x7fcbe419ba30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.443+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 >> 172.21.15.16:0/3723401876 conn(0x7fcbe4102050 msgr2=0x7fcbe4102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.443+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 shutdown_connections 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-25T01:18:08.443+0000 7fcbe8fa7700 1 -- 172.21.15.16:0/3723401876 wait complete. 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T01:18:09.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:09.037 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-04-25T01:18:09.078 INFO:tasks.cephadm:Fetching config... 2024-04-25T01:18:09.079 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:18:09.079 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T01:18:09.096 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T01:18:09.096 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:18:09.096 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T01:18:09.152 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T01:18:09.152 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:18:09.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T01:18:09.218 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T01:18:09.219 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:18:09.219 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T01:18:09.275 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T01:18:09.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:18:09.349 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:09.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:18:09.401 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:09.413 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:18:09.452 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqdk2vCNWHEX+3lEW42TPBlmtaGv72v2CaDqst7jrKqnlUK/yJbONf/Rf3Z/+DbFLK5hxrNcsSiTzJQCjNIJU2+cm+JQRHqHPkvhi7WYPF4m1dvLpiHNj1D/vWGkUxdhOHvlwoEdGI7snvVBpZqjYcj1x3suYnbSmxqR4zSwVMr9jQc9/dWSDQs1qYOuISmdYrtV9F8CObD1do2P0NPxy3mTTETKA/Ei+yUbq5Ri7Vv5qY63UrRr0dbJHMf8zzUQRMTmrBIcGXwV9OuMGInBeLUvrEiANW2Q/YNKE8IH0P8ETJKNvRCE8unbVXe0OidPkJa574utthzZML5zImCZ5bkO8nAkccYmajr5V8nQJztEL1r8kcaVVHKdCijHYcLDBrnRiskIfh9ExQNfaCyY/ZwGIaBxGLOKve/wyqyqZjRUhE/tDAYrGI79mehdD2ZJYZU07MTCq8nhAGbw+vj8eSP8pCi2OineMbP/ZedQusa6TkmyTDSebwEB5Hb3gZws0= ceph-605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:09.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T01:18:09.593 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:09 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3723401876' entity='client.admin' 2024-04-25T01:18:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:11.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.976+0000 7fc43a55f700 1 -- 172.21.15.16:0/1215089904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 msgr2=0x7fc434104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 --2- 172.21.15.16:0/1215089904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434104bc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc424009910 tx=0x7fc424009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 -- 172.21.15.16:0/1215089904 shutdown_connections 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 --2- 172.21.15.16:0/1215089904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434104bc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 -- 172.21.15.16:0/1215089904 >> 172.21.15.16:0/1215089904 conn(0x7fc4340fc180 msgr2=0x7fc4340fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 -- 172.21.15.16:0/1215089904 shutdown_connections 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 -- 172.21.15.16:0/1215089904 wait complete. 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 Processor -- start 2024-04-25T01:18:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.977+0000 7fc43a55f700 1 -- start start 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc43a55f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc43a55f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc434107be0 con 0x7fc4341027d0 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc433fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc433fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49612/0 (socket says 172.21.15.16:49612) 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc433fff700 1 -- 172.21.15.16:0/4073006289 learned_addr learned my addr 172.21.15.16:0/4073006289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc433fff700 1 -- 172.21.15.16:0/4073006289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc434197850 con 0x7fc4341027d0 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc433fff700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434197310 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc424005b40 tx=0x7fc424004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424027070 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc42400be10 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc434197ae0 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc424022de0 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.978+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43419a780 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.979+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc42402a070 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.979+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434108360 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.979+0000 7fc4317fa700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc420039150 0x7fc42003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.980+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc42401e030 con 0x7fc4341027d0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.980+0000 7fc4337fe700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc420039150 0x7fc42003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:11.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.980+0000 7fc4337fe700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc420039150 0x7fc42003b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc41c006fd0 tx=0x7fc41c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:11.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:11.985+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc424013070 con 0x7fc4341027d0 2024-04-25T01:18:12.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.207+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fc43402dc40 con 0x7fc4341027d0 2024-04-25T01:18:12.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.209+0000 7fc4317fa700 1 -- 172.21.15.16:0/4073006289 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fc4240292c0 con 0x7fc4341027d0 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc420039150 msgr2=0x7fc42003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc420039150 0x7fc42003b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc41c006fd0 tx=0x7fc41c006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 msgr2=0x7fc434197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434197310 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc424005b40 tx=0x7fc424004f80 comp rx=0 tx=0).stop 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 shutdown_connections 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc420039150 0x7fc42003b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 --2- 172.21.15.16:0/4073006289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4341027d0 0x7fc434197310 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 >> 172.21.15.16:0/4073006289 conn(0x7fc4340fc180 msgr2=0x7fc4340fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.211+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 shutdown_connections 2024-04-25T01:18:12.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:12.212+0000 7fc43a55f700 1 -- 172.21.15.16:0/4073006289 wait complete. 2024-04-25T01:18:12.777 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T01:18:12.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T01:18:13.027 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:13.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:13 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4073006289' entity='client.admin' 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.361+0000 7f737e9c8700 1 -- 172.21.15.16:0/2421496383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 msgr2=0x7f73781020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.361+0000 7f737e9c8700 1 --2- 172.21.15.16:0/2421496383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f73781020f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7368009910 tx=0x7f7368009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 -- 172.21.15.16:0/2421496383 shutdown_connections 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 --2- 172.21.15.16:0/2421496383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f73781020f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 -- 172.21.15.16:0/2421496383 >> 172.21.15.16:0/2421496383 conn(0x7f73780fd230 msgr2=0x7f73780ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 -- 172.21.15.16:0/2421496383 shutdown_connections 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 -- 172.21.15.16:0/2421496383 wait complete. 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 Processor -- start 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.362+0000 7f737e9c8700 1 -- start start 2024-04-25T01:18:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f737e9c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f7378075ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f737e9c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7378107c10 con 0x7f7378101cd0 2024-04-25T01:18:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f7377fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f7378075ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f7377fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f7378075ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49636/0 (socket says 172.21.15.16:49636) 2024-04-25T01:18:15.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f7377fff700 1 -- 172.21.15.16:0/2870633201 learned_addr learned my addr 172.21.15.16:0/2870633201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f7377fff700 1 -- 172.21.15.16:0/2870633201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73780763e0 con 0x7f7378101cd0 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f7377fff700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f7378075ea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7368005950 tx=0x7f7368004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7368026070 con 0x7f7378101cd0 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f736800bc00 con 0x7f7378101cd0 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73680039a0 con 0x7f7378101cd0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7378074620 con 0x7f7378101cd0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.363+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7378074b00 con 0x7f7378101cd0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.364+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7368029070 con 0x7f7378101cd0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.364+0000 7f73757fa700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7360039150 0x7f736003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.364+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f736801e070 con 0x7f7378101cd0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.364+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7378108390 con 0x7f7378101cd0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.364+0000 7f73777fe700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7360039150 0x7f736003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.365+0000 7f73777fe700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7360039150 0x7f736003b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f736c006fd0 tx=0x7f736c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.370+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7368030070 con 0x7f7378101cd0 2024-04-25T01:18:15.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.582+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f7378190600 con 0x7f7360039150 2024-04-25T01:18:15.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:15.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:15.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:15.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:15.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:15.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:15.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.585+0000 7f73757fa700 1 -- 172.21.15.16:0/2870633201 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7378190600 con 0x7f7360039150 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.587+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7360039150 msgr2=0x7f736003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.587+0000 7f737e9c8700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7360039150 0x7f736003b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f736c006fd0 tx=0x7f736c006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.587+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 msgr2=0x7f7378075ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.587+0000 7f737e9c8700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f7378075ea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7368005950 tx=0x7f7368004fa0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.588+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 shutdown_connections 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.588+0000 7f737e9c8700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7360039150 0x7f736003b610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.588+0000 7f737e9c8700 1 --2- 172.21.15.16:0/2870633201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7378101cd0 0x7f7378075ea0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.588+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 >> 172.21.15.16:0/2870633201 conn(0x7f73780fd230 msgr2=0x7f73780fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.588+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 shutdown_connections 2024-04-25T01:18:15.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:15.588+0000 7f737e9c8700 1 -- 172.21.15.16:0/2870633201 wait complete. 2024-04-25T01:18:16.338 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi038 2024-04-25T01:18:16.338 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:18:16.338 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:18:16.358 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:18:16.358 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:16.416 INFO:tasks.cephadm:Adding host smithi038 to orchestrator... 2024-04-25T01:18:16.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch host add smithi038 2024-04-25T01:18:16.661 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:16.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:16.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: Updating smithi016:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: Updating smithi016:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 -- 172.21.15.16:0/1290068712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 msgr2=0x7f4b6c1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 --2- 172.21.15.16:0/1290068712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1022b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009910 tx=0x7f4b68009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 -- 172.21.15.16:0/1290068712 shutdown_connections 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 --2- 172.21.15.16:0/1290068712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1022b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 -- 172.21.15.16:0/1290068712 >> 172.21.15.16:0/1290068712 conn(0x7f4b6c0fd760 msgr2=0x7f4b6c0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 -- 172.21.15.16:0/1290068712 shutdown_connections 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 -- 172.21.15.16:0/1290068712 wait complete. 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 Processor -- start 2024-04-25T01:18:18.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.312+0000 7f4b740d2700 1 -- start start 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b740d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b740d2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c10c150 con 0x7f4b6c101ed0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b71e6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b71e6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49658/0 (socket says 172.21.15.16:49658) 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b71e6e700 1 -- 172.21.15.16:0/3647674685 learned_addr learned my addr 172.21.15.16:0/3647674685 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b71e6e700 1 -- 172.21.15.16:0/3647674685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c197bf0 con 0x7f4b6c101ed0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b71e6e700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1976b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006010 tx=0x7f4b68005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b68026070 con 0x7f4b6c101ed0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b6800bba0 con 0x7f4b6c101ed0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b680038a0 con 0x7f4b6c101ed0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.313+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c197ee0 con 0x7f4b6c101ed0 2024-04-25T01:18:18.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.314+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c19bf80 con 0x7f4b6c101ed0 2024-04-25T01:18:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.314+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4b68029070 con 0x7f4b6c101ed0 2024-04-25T01:18:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.314+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c1033b0 con 0x7f4b6c101ed0 2024-04-25T01:18:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.315+0000 7f4b62ffd700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b58039150 0x7f4b5803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.315+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4b6801e070 con 0x7f4b6c101ed0 2024-04-25T01:18:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.315+0000 7f4b7166d700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b58039150 0x7f4b5803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.315+0000 7f4b7166d700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b58039150 0x7f4b5803b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006fd0 tx=0x7f4b5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.319+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4b68013070 con 0x7f4b6c101ed0 2024-04-25T01:18:18.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:18.530+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}) -- 0x7f4b6c198a10 con 0x7f4b58039150 2024-04-25T01:18:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:19 smithi016 ceph-mon[29153]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:19 smithi016 ceph-mon[29153]: Deploying cephadm binary to smithi038 2024-04-25T01:18:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:20 smithi016 ceph-mon[29153]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:22.780 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi038' with addr '172.21.15.38' 2024-04-25T01:18:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.796+0000 7f4b62ffd700 1 -- 172.21.15.16:0/3647674685 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4b6c198a10 con 0x7f4b58039150 2024-04-25T01:18:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b58039150 msgr2=0x7f4b5803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b58039150 0x7f4b5803b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006fd0 tx=0x7f4b5c006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 msgr2=0x7f4b6c1976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1976b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006010 tx=0x7f4b68005780 comp rx=0 tx=0).stop 2024-04-25T01:18:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 shutdown_connections 2024-04-25T01:18:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b58039150 0x7f4b5803b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 --2- 172.21.15.16:0/3647674685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6c101ed0 0x7f4b6c1976b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 >> 172.21.15.16:0/3647674685 conn(0x7f4b6c0fd760 msgr2=0x7f4b6c1038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 shutdown_connections 2024-04-25T01:18:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:22.799+0000 7f4b740d2700 1 -- 172.21.15.16:0/3647674685 wait complete. 2024-04-25T01:18:22.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:22 smithi016 ceph-mon[29153]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:23.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:18:23.653 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:24.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:24.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:23 smithi016 ceph-mon[29153]: Added host smithi038 2024-04-25T01:18:24.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:24.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.143+0000 7f47ee269700 1 -- 172.21.15.16:0/3774885343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 msgr2=0x7f47e8102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.143+0000 7f47ee269700 1 --2- 172.21.15.16:0/3774885343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8102280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009910 tx=0x7f47d8009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 -- 172.21.15.16:0/3774885343 shutdown_connections 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 --2- 172.21.15.16:0/3774885343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8102280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 -- 172.21.15.16:0/3774885343 >> 172.21.15.16:0/3774885343 conn(0x7f47e80fd770 msgr2=0x7f47e80ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 -- 172.21.15.16:0/3774885343 shutdown_connections 2024-04-25T01:18:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 -- 172.21.15.16:0/3774885343 wait complete. 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 Processor -- start 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.144+0000 7f47ee269700 1 -- start start 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47ee269700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47ee269700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f47e810c120 con 0x7f47e8101ea0 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37940/0 (socket says 172.21.15.16:37940) 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e7fff700 1 -- 172.21.15.16:0/1770390314 learned_addr learned my addr 172.21.15.16:0/1770390314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e7fff700 1 -- 172.21.15.16:0/1770390314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e8076420 con 0x7f47e8101ea0 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e7fff700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8074530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f47d80056d0 tx=0x7f47d80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47d8026070 con 0x7f47e8101ea0 2024-04-25T01:18:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47d800bc00 con 0x7f47e8101ea0 2024-04-25T01:18:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47d80039b0 con 0x7f47e8101ea0 2024-04-25T01:18:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e8074ad0 con 0x7f47e8101ea0 2024-04-25T01:18:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.145+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e8074f90 con 0x7f47e8101ea0 2024-04-25T01:18:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.146+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f47d8029070 con 0x7f47e8101ea0 2024-04-25T01:18:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.146+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e8103380 con 0x7f47e8101ea0 2024-04-25T01:18:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.146+0000 7f47e57fa700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f47d0039150 0x7f47d003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.146+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f47d801e070 con 0x7f47e8101ea0 2024-04-25T01:18:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.147+0000 7f47e77fe700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f47d0039150 0x7f47d003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.147+0000 7f47e77fe700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f47d0039150 0x7f47d003b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f47dc006fd0 tx=0x7f47dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.151+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f47d8013070 con 0x7f47e8101ea0 2024-04-25T01:18:25.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.363+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f47e8075220 con 0x7f47d0039150 2024-04-25T01:18:25.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:24 smithi016 ceph-mon[29153]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:25.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:25.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.364+0000 7f47e57fa700 1 -- 172.21.15.16:0/1770390314 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f47e8075220 con 0x7f47d0039150 2024-04-25T01:18:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:25.348 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}] 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f47d0039150 msgr2=0x7f47d003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f47d0039150 0x7f47d003b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f47dc006fd0 tx=0x7f47dc006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 msgr2=0x7f47e8074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8074530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f47d80056d0 tx=0x7f47d80057b0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 shutdown_connections 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f47d0039150 0x7f47d003b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 --2- 172.21.15.16:0/1770390314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47e8101ea0 0x7f47e8074530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 >> 172.21.15.16:0/1770390314 conn(0x7f47e80fd770 msgr2=0x7f47e8103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 shutdown_connections 2024-04-25T01:18:25.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:25.367+0000 7f47ee269700 1 -- 172.21.15.16:0/1770390314 wait complete. 2024-04-25T01:18:26.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:25 smithi016 ceph-mon[29153]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:18:26.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:25 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:26.031 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2024-04-25T01:18:26.032 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:18:26.032 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:18:26.050 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:18:26.050 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:26.108 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2024-04-25T01:18:26.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch host add smithi156 2024-04-25T01:18:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:27.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:26 smithi016 ceph-mon[29153]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 -- 172.21.15.16:0/668975116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 msgr2=0x7f732c1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 --2- 172.21.15.16:0/668975116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1011c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f731c009910 tx=0x7f731c009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 -- 172.21.15.16:0/668975116 shutdown_connections 2024-04-25T01:18:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 --2- 172.21.15.16:0/668975116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1011c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 -- 172.21.15.16:0/668975116 >> 172.21.15.16:0/668975116 conn(0x7f732c0fc690 msgr2=0x7f732c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 -- 172.21.15.16:0/668975116 shutdown_connections 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.870+0000 7f73319f6700 1 -- 172.21.15.16:0/668975116 wait complete. 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.871+0000 7f73319f6700 1 Processor -- start 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.871+0000 7f73319f6700 1 -- start start 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.871+0000 7f73319f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.871+0000 7f73319f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f732c109f50 con 0x7f732c100de0 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f732affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f732affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37958/0 (socket says 172.21.15.16:37958) 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f732affd700 1 -- 172.21.15.16:0/3289961697 learned_addr learned my addr 172.21.15.16:0/3289961697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f732affd700 1 -- 172.21.15.16:0/3289961697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c1959f0 con 0x7f732c100de0 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f732affd700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1954b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f731c0056d0 tx=0x7f731c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f731c026070 con 0x7f732c100de0 2024-04-25T01:18:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f731c00bba0 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732c195ce0 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f731c0037f0 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.872+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732c199d80 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.873+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f731c029070 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.873+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f732c1022c0 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.873+0000 7f73309f4700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f731403b350 0x7f731403d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.873+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f731c01e070 con 0x7f732c100de0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.874+0000 7f732a7fc700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f731403b350 0x7f731403d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.874+0000 7f732a7fc700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f731403b350 0x7f731403d810 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7320006fd0 tx=0x7f7320006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:27.878+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f731c013070 con 0x7f732c100de0 2024-04-25T01:18:28.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:28.089+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) -- 0x7f732c196810 con 0x7f731403b350 2024-04-25T01:18:29.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:29.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:30.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:29 smithi016 ceph-mon[29153]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-25T01:18:30.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:29 smithi016 ceph-mon[29153]: Deploying cephadm binary to smithi156 2024-04-25T01:18:30.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:29 smithi016 ceph-mon[29153]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:30.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:29 smithi016 ceph-mon[29153]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:31.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:30 smithi016 ceph-mon[29153]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:32.305 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi156' with addr '172.21.15.156' 2024-04-25T01:18:32.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.321+0000 7f73309f4700 1 -- 172.21.15.16:0/3289961697 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f732c196810 con 0x7f731403b350 2024-04-25T01:18:32.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f731403b350 msgr2=0x7f731403d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f731403b350 0x7f731403d810 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7320006fd0 tx=0x7f7320006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:32.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 msgr2=0x7f732c1954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1954b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f731c0056d0 tx=0x7f731c0057b0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 shutdown_connections 2024-04-25T01:18:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f731403b350 0x7f731403d810 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 --2- 172.21.15.16:0/3289961697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f732c100de0 0x7f732c1954b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 >> 172.21.15.16:0/3289961697 conn(0x7f732c0fc690 msgr2=0x7f732c1027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 shutdown_connections 2024-04-25T01:18:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:32.324+0000 7f73319f6700 1 -- 172.21.15.16:0/3289961697 wait complete. 2024-04-25T01:18:33.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:33.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:33.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:33.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:33.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:33.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:33.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: Added host smithi156 2024-04-25T01:18:33.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:33.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:32 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:33.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:18:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:34.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 -- 172.21.15.16:0/3939112446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 msgr2=0x7f8b48069a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 --2- 172.21.15.16:0/3939112446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b48069a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009910 tx=0x7f8b44009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:34.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 -- 172.21.15.16:0/3939112446 shutdown_connections 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 --2- 172.21.15.16:0/3939112446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b48069a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 -- 172.21.15.16:0/3939112446 >> 172.21.15.16:0/3939112446 conn(0x7f8b48076690 msgr2=0x7f8b48076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 -- 172.21.15.16:0/3939112446 shutdown_connections 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.002+0000 7f8b4f374700 1 -- 172.21.15.16:0/3939112446 wait complete. 2024-04-25T01:18:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4f374700 1 Processor -- start 2024-04-25T01:18:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4f374700 1 -- start start 2024-04-25T01:18:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4f374700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b4819f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:34.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4f374700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810e380 con 0x7f8b48069650 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4d110700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b4819f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4d110700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b4819f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43596/0 (socket says 172.21.15.16:43596) 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.003+0000 7f8b4d110700 1 -- 172.21.15.16:0/3506965727 learned_addr learned my addr 172.21.15.16:0/3506965727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b4d110700 1 -- 172.21.15.16:0/3506965727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4819fba0 con 0x7f8b48069650 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b4d110700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b4819f660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b440056d0 tx=0x7f8b440057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b44026070 con 0x7f8b48069650 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b4400bc00 con 0x7f8b48069650 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4819fe30 con 0x7f8b48069650 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b440039b0 con 0x7f8b48069650 2024-04-25T01:18:34.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.004+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b48199730 con 0x7f8b48069650 2024-04-25T01:18:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.005+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8b44029070 con 0x7f8b48069650 2024-04-25T01:18:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.005+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b48101160 con 0x7f8b48069650 2024-04-25T01:18:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.005+0000 7f8b3e7fc700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8b34039150 0x7f8b3403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.005+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b4401e070 con 0x7f8b48069650 2024-04-25T01:18:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.005+0000 7f8b4c90f700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8b34039150 0x7f8b3403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:34.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.006+0000 7f8b4c90f700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8b34039150 0x7f8b3403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8b38006fd0 tx=0x7f8b38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:34.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.010+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8b44013070 con 0x7f8b48069650 2024-04-25T01:18:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.220+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b48063890 con 0x7f8b34039150 2024-04-25T01:18:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.221+0000 7f8b3e7fc700 1 -- 172.21.15.16:0/3506965727 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f8b48063890 con 0x7f8b34039150 2024-04-25T01:18:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:18:35.205 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8b34039150 msgr2=0x7f8b3403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8b34039150 0x7f8b3403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8b38006fd0 tx=0x7f8b38006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 msgr2=0x7f8b4819f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b4819f660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b440056d0 tx=0x7f8b440057b0 comp rx=0 tx=0).stop 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 shutdown_connections 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8b34039150 0x7f8b3403b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 --2- 172.21.15.16:0/3506965727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b48069650 0x7f8b4819f660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.223+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 >> 172.21.15.16:0/3506965727 conn(0x7f8b48076690 msgr2=0x7f8b481091b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.224+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 shutdown_connections 2024-04-25T01:18:35.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:35.224+0000 7f8b4f374700 1 -- 172.21.15.16:0/3506965727 wait complete. 2024-04-25T01:18:35.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:35 smithi016 ceph-mon[29153]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:35.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:35 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:35.840 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T01:18:35.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T01:18:36.087 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:18:36.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:36 smithi016 ceph-mon[29153]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:18:37.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:37 smithi016 ceph-mon[29153]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.657+0000 7f30acb16700 1 -- 172.21.15.16:0/360381366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 msgr2=0x7f30a8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.657+0000 7f30acb16700 1 --2- 172.21.15.16:0/360381366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a8101180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3098009910 tx=0x7f3098009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 -- 172.21.15.16:0/360381366 shutdown_connections 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 --2- 172.21.15.16:0/360381366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a8101180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 -- 172.21.15.16:0/360381366 >> 172.21.15.16:0/360381366 conn(0x7f30a80fc630 msgr2=0x7f30a80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 -- 172.21.15.16:0/360381366 shutdown_connections 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 -- 172.21.15.16:0/360381366 wait complete. 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 Processor -- start 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.658+0000 7f30acb16700 1 -- start start 2024-04-25T01:18:37.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30acb16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30acb16700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30a810b020 con 0x7f30a8100da0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30a659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30a659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a819f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43610/0 (socket says 172.21.15.16:43610) 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30a659c700 1 -- 172.21.15.16:0/1317401700 learned_addr learned my addr 172.21.15.16:0/1317401700 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30a659c700 1 -- 172.21.15.16:0/1317401700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a819fb50 con 0x7f30a8100da0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30a659c700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a819f610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f30980056d0 tx=0x7f30980057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3098026070 con 0x7f30a8100da0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a819fe40 con 0x7f30a8100da0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f309800bba0 con 0x7f30a8100da0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.659+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30980037f0 con 0x7f30a8100da0 2024-04-25T01:18:37.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.660+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a81998b0 con 0x7f30a8100da0 2024-04-25T01:18:37.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.660+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3098029070 con 0x7f30a8100da0 2024-04-25T01:18:37.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.660+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a8102280 con 0x7f30a8100da0 2024-04-25T01:18:37.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.661+0000 7f30977fe700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f30900390f0 0x7f309003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.661+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f309801e070 con 0x7f30a8100da0 2024-04-25T01:18:37.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.661+0000 7f30a5d9b700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f30900390f0 0x7f309003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.661+0000 7f30a5d9b700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f30900390f0 0x7f309003b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.666+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3098013070 con 0x7f30a8100da0 2024-04-25T01:18:37.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:37.878+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f30a80008d0 con 0x7f30a8100da0 2024-04-25T01:18:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.826+0000 7f30977fe700 1 -- 172.21.15.16:0/1317401700 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f3098030090 con 0x7f30a8100da0 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f30900390f0 msgr2=0x7f309003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f30900390f0 0x7f309003b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 msgr2=0x7f30a819f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a819f610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f30980056d0 tx=0x7f30980057b0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 shutdown_connections 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f30900390f0 0x7f309003b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 --2- 172.21.15.16:0/1317401700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a8100da0 0x7f30a819f610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.828+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 >> 172.21.15.16:0/1317401700 conn(0x7f30a80fc630 msgr2=0x7f30a806a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.829+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 shutdown_connections 2024-04-25T01:18:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:18:38.829+0000 7f30acb16700 1 -- 172.21.15.16:0/1317401700 wait complete. 2024-04-25T01:18:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-04-25T01:18:39.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1317401700' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:39.425 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-04-25T01:18:39.425 INFO:tasks.cephadm:Adding mon.b on smithi038 2024-04-25T01:18:39.425 INFO:tasks.cephadm:Adding mon.c on smithi156 2024-04-25T01:18:39.425 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c' 2024-04-25T01:18:39.650 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:40.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:39 smithi016 ceph-mon[29153]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:40.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:39 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1317401700' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:18:40.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:39 smithi016 ceph-mon[29153]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:18:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:40 smithi016 ceph-mon[29153]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:41.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.152+0000 7fe679764700 1 -- 172.21.15.156:0/4089346758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 msgr2=0x7fe674101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.152+0000 7fe679764700 1 --2- 172.21.15.156:0/4089346758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674101180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009910 tx=0x7fe65c009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:41.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.152+0000 7fe679764700 1 -- 172.21.15.156:0/4089346758 shutdown_connections 2024-04-25T01:18:41.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.152+0000 7fe679764700 1 --2- 172.21.15.156:0/4089346758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674101180 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.152+0000 7fe679764700 1 -- 172.21.15.156:0/4089346758 >> 172.21.15.156:0/4089346758 conn(0x7fe6740fc610 msgr2=0x7fe6740fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.153+0000 7fe679764700 1 -- 172.21.15.156:0/4089346758 shutdown_connections 2024-04-25T01:18:41.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.153+0000 7fe679764700 1 -- 172.21.15.156:0/4089346758 wait complete. 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.153+0000 7fe679764700 1 Processor -- start 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.153+0000 7fe679764700 1 -- start start 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.153+0000 7fe679764700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.153+0000 7fe679764700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe67410b020 con 0x7fe674100da0 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe672ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe672ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.156:44238/0 (socket says 172.21.15.156:44238) 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe672ffd700 1 -- 172.21.15.156:0/144590217 learned_addr learned my addr 172.21.15.156:0/144590217 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe672ffd700 1 -- 172.21.15.156:0/144590217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe67406af00 con 0x7fe674100da0 2024-04-25T01:18:41.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe672ffd700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674196320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0056d0 tx=0x7fe65c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe65c026070 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe674196860 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe65c00bba0 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.154+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe65c0037f0 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.155+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe674196d20 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.155+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe65c029070 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.155+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe674102280 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.156+0000 7fe66bfff700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe6600390f0 0x7fe66003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.156+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe65c01e070 con 0x7fe674100da0 2024-04-25T01:18:41.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.156+0000 7fe6727fc700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe6600390f0 0x7fe66003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.156+0000 7fe6727fc700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe6600390f0 0x7fe66003b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe664006fd0 tx=0x7fe664006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.161+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe65c013070 con 0x7fe674100da0 2024-04-25T01:18:41.356 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.373+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}) -- 0x7fe67406a760 con 0x7fe6600390f0 2024-04-25T01:18:41.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.379+0000 7fe66bfff700 1 -- 172.21.15.156:0/144590217 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe67406a760 con 0x7fe6600390f0 2024-04-25T01:18:41.362 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mon update... 2024-04-25T01:18:41.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe6600390f0 msgr2=0x7fe66003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe6600390f0 0x7fe66003b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe664006fd0 tx=0x7fe664006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:41.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 msgr2=0x7fe674196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674196320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0056d0 tx=0x7fe65c0057b0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 shutdown_connections 2024-04-25T01:18:41.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe6600390f0 0x7fe66003b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 --2- 172.21.15.156:0/144590217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe674100da0 0x7fe674196320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.381+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 >> 172.21.15.156:0/144590217 conn(0x7fe6740fc610 msgr2=0x7fe674106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.382+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 shutdown_connections 2024-04-25T01:18:41.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:41.382+0000 7fe679764700 1 -- 172.21.15.156:0/144590217 wait complete. 2024-04-25T01:18:41.973 DEBUG:teuthology.orchestra.run.smithi038:mon.b> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T01:18:41.976 DEBUG:teuthology.orchestra.run.smithi156:mon.c> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T01:18:41.978 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:18:41.979 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:18:42.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c;count:3 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:42.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:42 smithi016 ceph-mon[29153]: Deploying daemon mon.c on smithi156 2024-04-25T01:18:42.646 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:43.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:43 smithi016 ceph-mon[29153]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.724+0000 7fa504cce700 1 -- 172.21.15.156:0/1956058550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 msgr2=0x7fa500108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.724+0000 7fa504cce700 1 --2- 172.21.15.156:0/1956058550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa500108910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0009910 tx=0x7fa4f0009c20 comp rx=0 tx=0).stop 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.724+0000 7fa504cce700 1 -- 172.21.15.156:0/1956058550 shutdown_connections 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.724+0000 7fa504cce700 1 --2- 172.21.15.156:0/1956058550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa500108910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.724+0000 7fa504cce700 1 -- 172.21.15.156:0/1956058550 >> 172.21.15.156:0/1956058550 conn(0x7fa500077ac0 msgr2=0x7fa500079ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.725+0000 7fa504cce700 1 -- 172.21.15.156:0/1956058550 shutdown_connections 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.725+0000 7fa504cce700 1 -- 172.21.15.156:0/1956058550 wait complete. 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.725+0000 7fa504cce700 1 Processor -- start 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.725+0000 7fa504cce700 1 -- start start 2024-04-25T01:18:45.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa504cce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa5001a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa504cce700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa500112750 con 0x7fa500108530 2024-04-25T01:18:45.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa4fe59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa5001a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa4fe59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa5001a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.156:58818/0 (socket says 172.21.15.156:58818) 2024-04-25T01:18:45.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa4fe59c700 1 -- 172.21.15.156:0/749571212 learned_addr learned my addr 172.21.15.156:0/749571212 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:18:45.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa4fe59c700 1 -- 172.21.15.156:0/749571212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5001a3e10 con 0x7fa500108530 2024-04-25T01:18:45.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.726+0000 7fa4fe59c700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa5001a38d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00056d0 tx=0x7fa4f00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:45.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.727+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4f0026070 con 0x7fa500108530 2024-04-25T01:18:45.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.727+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4f000bba0 con 0x7fa500108530 2024-04-25T01:18:45.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.727+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4f00037f0 con 0x7fa500108530 2024-04-25T01:18:45.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.727+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5001a4100 con 0x7fa500108530 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.727+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50019da90 con 0x7fa500108530 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.728+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa4f0029070 con 0x7fa500108530 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.728+0000 7fa4ef7fe700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4e80390f0 0x7fa4e803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.728+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa4f001e070 con 0x7fa500108530 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.728+0000 7fa4fdd9b700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4e80390f0 0x7fa4e803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.728+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5001099b0 con 0x7fa500108530 2024-04-25T01:18:45.712 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.729+0000 7fa4fdd9b700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4e80390f0 0x7fa4e803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4006fd0 tx=0x7fa4f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:45.716 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:45.734+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa4f0030070 con 0x7fa500108530 2024-04-25T01:18:45.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:45 smithi016 ceph-mon[29153]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.047+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa5000638c0 con 0x7fa500108530 2024-04-25T01:18:46.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.048+0000 7fa4ef7fe700 1 -- 172.21.15.156:0/749571212 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fa5000638c0 con 0x7fa500108530 2024-04-25T01:18:46.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:18:46.031 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","modified":"2024-04-25T01:16:49.048092Z","created":"2024-04-25T01:16:49.048092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T01:18:46.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4e80390f0 msgr2=0x7fa4e803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:46.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4e80390f0 0x7fa4e803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4006fd0 tx=0x7fa4f4006e40 comp rx=0 tx=0).stop 2024-04-25T01:18:46.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 msgr2=0x7fa5001a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:46.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa5001a38d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00056d0 tx=0x7fa4f00057b0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 shutdown_connections 2024-04-25T01:18:46.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4e80390f0 0x7fa4e803b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 --2- 172.21.15.156:0/749571212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500108530 0x7fa5001a38d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 >> 172.21.15.156:0/749571212 conn(0x7fa500077ac0 msgr2=0x7fa500109ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:46.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.050+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 shutdown_connections 2024-04-25T01:18:46.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:46.051+0000 7fa504cce700 1 -- 172.21.15.156:0/749571212 wait complete. 2024-04-25T01:18:46.034 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2024-04-25T01:18:46.142 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 podman[32599]: 2024-04-25 01:18:46.136035978 +0000 UTC m=+0.352723583 container start d84995c16b88fdbfac3b4663819ce861223d4cb1a281bc9cbf58be1d064f649f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T01:18:46.142 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:18:46.143 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:18:46.143 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pidfile_write: ignore empty --pid-file 2024-04-25T01:18:46.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: load: jerasure load: lrc 2024-04-25T01:18:46.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:18:46.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Git sha 0 2024-04-25T01:18:46.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:18:46.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: DB SUMMARY 2024-04-25T01:18:46.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: DB Session ID: NYRGTMCUBF8LY7PS2TKY 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.env: 0x55e00b5f7c60 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:18:46.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.info_log: 0x55e00c31baa0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.statistics: (nil) 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.use_fsync: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:18:46.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:18:46.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.db_log_dir: 2024-04-25T01:18:46.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.wal_dir: 2024-04-25T01:18:46.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:18:46.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:18:46.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:18:46.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:18:46.508 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:18:46.508 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.write_buffer_manager: 0x55e00c2ee1e0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.unordered_write: 0 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:18:46.509 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.row_cache: None 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.wal_filter: None 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.wal_compression: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:18:46.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:18:46.511 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_open_files: -1 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Compression algorithms supported: 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kZSTD supported: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:18:46.512 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.merge_operator: 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_filter: None 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e00c31bba0) 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:18:46.513 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_type: 0 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_index_type: 0 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_shortening: 1 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: checksum: 4 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: no_block_cache: 0 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache: 0x55e00c31f090 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_options: 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: capacity : 536870912 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_shard_bits : 4 2024-04-25T01:18:46.514 INFO:journalctl@ceph.mon.c.smithi156.stdout: strict_capacity_limit : 0 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_compressed: (nil) 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: persistent_cache: (nil) 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size: 4096 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size_deviation: 10 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_restart_interval: 16 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_block_restart_interval: 1 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: metadata_block_size: 4096 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: partition_filters: 0 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: use_delta_encoding: 1 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: filter_policy: bloomfilter 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: whole_key_filtering: 1 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: verify_compression: 0 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: format_version: 5 2024-04-25T01:18:46.515 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_index_compression: 1 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_align: 0 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout: max_auto_readahead_size: 262144 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout: prepopulate_block_cache: 0 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression: NoCompression 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.num_levels: 7 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:18:46.516 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:18:46.517 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:18:46.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:18:46.519 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:18:46.520 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.ttl: 2592000 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:18:46.521 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 29a7cfdf-e7be-466a-a925-6e8f71e2a0a9 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007926166352, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007926166771, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714007926, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "29a7cfdf-e7be-466a-a925-6e8f71e2a0a9", "db_session_id": "NYRGTMCUBF8LY7PS2TKY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007926166851, "job": 1, "event": "recovery_finished"} 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T01:18:46.522 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e00c410000 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: DB pointer 0x55e00c3fa000 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** DB Stats ** 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T01:18:46.523 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T01:18:46.524 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache BinnedLRUCache@0x55e00c31f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-25T01:18:46.525 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: starting mon.c rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(???) e0 preinit fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout: e1 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout: legacy client fscid: -1 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout: No filesystems configured 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:18:46.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.527 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Deploying cephadm binary to smithi156 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.528 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Added host smithi156 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.529 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1317401700' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1317401700' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.530 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c;count:3 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: Deploying daemon mon.c on smithi156 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 ceph-mon[32631]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 bash[32599]: d84995c16b88fdbfac3b4663819ce861223d4cb1a281bc9cbf58be1d064f649f 2024-04-25T01:18:46.531 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:46 smithi156 systemd[1]: Started Ceph mon.c for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:18:47.839 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:18:47.839 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:18:48.086 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:18:51.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:51.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: mon.a calling monitor election 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: mon.c calling monitor election 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: monmap epoch 2 2024-04-25T01:18:51.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: last_changed 2024-04-25T01:18:46.190029+0000 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: min_mon_release 19 (squid) 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: election_strategy: 1 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: fsmap 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: mgrmap e13: x(active, since 51s) 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: overall HEALTH_OK 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:51.506 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:18:51.507 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:51.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: mon.a calling monitor election 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: mon.c calling monitor election 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:18:51.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: monmap epoch 2 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: last_changed 2024-04-25T01:18:46.190029+0000 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: min_mon_release 19 (squid) 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: election_strategy: 1 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: fsmap 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: mgrmap e13: x(active, since 51s) 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: overall HEALTH_OK 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:18:51.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:52.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:52 smithi156 ceph-mon[32631]: Deploying daemon mon.b on smithi038 2024-04-25T01:18:52.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.546+0000 7f837fe0a700 1 -- 172.21.15.156:0/3189323010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 msgr2=0x7f8364005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.546+0000 7f837fe0a700 1 --2- 172.21.15.156:0/3189323010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8364005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8378078450 tx=0x7f837400feb0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.546+0000 7f837fe0a700 1 -- 172.21.15.156:0/3189323010 shutdown_connections 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.546+0000 7f837fe0a700 1 --2- 172.21.15.156:0/3189323010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8364005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.546+0000 7f837fe0a700 1 -- 172.21.15.156:0/3189323010 >> 172.21.15.156:0/3189323010 conn(0x7f8378077110 msgr2=0x7f8378077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.547+0000 7f837fe0a700 1 -- 172.21.15.156:0/3189323010 shutdown_connections 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.547+0000 7f837fe0a700 1 -- 172.21.15.156:0/3189323010 wait complete. 2024-04-25T01:18:52.530 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.547+0000 7f837fe0a700 1 Processor -- start 2024-04-25T01:18:52.531 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837fe0a700 1 -- start start 2024-04-25T01:18:52.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837fe0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8378101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837fe0a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837fe0a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8378109c90 con 0x7f83640035c0 2024-04-25T01:18:52.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837fe0a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8378109b10 con 0x7f8378101b80 2024-04-25T01:18:52.532 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:33984/0 (socket says 172.21.15.156:33984) 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 -- 172.21.15.156:0/1123978964 learned_addr learned my addr 172.21.15.156:0/1123978964 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837dba6700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8378101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 -- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 msgr2=0x7f8378102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 -- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 msgr2=0x7f8378102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.548+0000 7f837d3a5700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.549+0000 7f837dba6700 1 -- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 msgr2=0x7f8378102000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.549+0000 7f837dba6700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.549+0000 7f837dba6700 1 -- 172.21.15.156:0/1123978964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8378103560 con 0x7f83640035c0 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.549+0000 7f837dba6700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8378101640 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8374000c00 tx=0x7f83740046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.549+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8374013b00 con 0x7f83640035c0 2024-04-25T01:18:52.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.550+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83781a3a30 con 0x7f83640035c0 2024-04-25T01:18:52.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.550+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83740044e0 con 0x7f83640035c0 2024-04-25T01:18:52.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.550+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83781a3f70 con 0x7f83640035c0 2024-04-25T01:18:52.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.550+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8374011470 con 0x7f83640035c0 2024-04-25T01:18:52.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.551+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8374023070 con 0x7f83640035c0 2024-04-25T01:18:52.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.551+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f837806a190 con 0x7f83640035c0 2024-04-25T01:18:52.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.551+0000 7f836effd700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f835c038e90 0x7f835c03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.551+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8374020050 con 0x7f83640035c0 2024-04-25T01:18:52.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.551+0000 7f837d3a5700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f835c038e90 0x7f835c03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.552+0000 7f837d3a5700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f835c038e90 0x7f835c03b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8368009180 tx=0x7f8368008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:52.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.556+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8374028020 con 0x7f83640035c0 2024-04-25T01:18:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:52 smithi016 ceph-mon[29153]: Deploying daemon mon.b on smithi038 2024-04-25T01:18:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:52.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.834+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f837802db30 con 0x7f83640035c0 2024-04-25T01:18:52.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.834+0000 7f836effd700 1 -- 172.21.15.156:0/1123978964 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f8374013ca0 con 0x7f83640035c0 2024-04-25T01:18:52.817 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:18:52.817 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","modified":"2024-04-25T01:18:46.190029Z","created":"2024-04-25T01:16:49.048092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f835c038e90 msgr2=0x7f835c03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f835c038e90 0x7f835c03b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8368009180 tx=0x7f8368008040 comp rx=0 tx=0).stop 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 msgr2=0x7f8378101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8378101640 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8374000c00 tx=0x7f83740046f0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 shutdown_connections 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f835c038e90 0x7f835c03b350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83640035c0 0x7f8378101640 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 --2- 172.21.15.156:0/1123978964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8378101b80 0x7f8378102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 >> 172.21.15.156:0/1123978964 conn(0x7f8378077110 msgr2=0x7f83780fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 shutdown_connections 2024-04-25T01:18:52.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:18:52.837+0000 7f837fe0a700 1 -- 172.21.15.156:0/1123978964 wait complete. 2024-04-25T01:18:52.821 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2024-04-25T01:18:53.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:53 smithi156 ceph-mon[32631]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:53.374 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:18:53 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/1123978964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:18:53.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:53 smithi016 ceph-mon[29153]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:53.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:18:53 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/1123978964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:18:54.401 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:18:54.402 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:18:54.647 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:18:55.096 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 podman[32822]: 2024-04-25 01:18:55.007533069 +0000 UTC m=+0.117814735 container create 97c40aa9d4810b99251505da6d09efd8a3c61619c42455529904d9f9931614be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 podman[32822]: 2024-04-25 01:18:55.193499421 +0000 UTC m=+0.303781089 container init 97c40aa9d4810b99251505da6d09efd8a3c61619c42455529904d9f9931614be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 podman[32822]: 2024-04-25 01:18:55.196493567 +0000 UTC m=+0.306775235 container start 97c40aa9d4810b99251505da6d09efd8a3c61619c42455529904d9f9931614be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pidfile_write: ignore empty --pid-file 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: load: jerasure load: lrc 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:18:55.470 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Git sha 0 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: DB SUMMARY 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: DB Session ID: R3LNU15GI7LAGWGXCDEJ 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:18:55.471 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:18:55.472 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:18:55.472 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.env: 0x55cb7d09dc60 2024-04-25T01:18:55.472 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:18:55.472 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.info_log: 0x55cb7db89aa0 2024-04-25T01:18:55.472 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.statistics: (nil) 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.use_fsync: 0 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:18:55.473 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.db_log_dir: 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.wal_dir: 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:18:55.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.write_buffer_manager: 0x55cb7db5c1e0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.unordered_write: 0 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:18:55.475 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.row_cache: None 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.wal_filter: None 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.wal_compression: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:18:55.476 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:18:55.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_open_files: -1 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Compression algorithms supported: 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kZSTD supported: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:18:55.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.merge_operator: 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_filter: None 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:18:55.479 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cb7db89ba0) 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_type: 0 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: data_block_index_type: 0 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_shortening: 1 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: checksum: 4 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: no_block_cache: 0 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache: 0x55cb7db8d090 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_options: 2024-04-25T01:18:55.480 INFO:journalctl@ceph.mon.b.smithi038.stdout: capacity : 536870912 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: num_shard_bits : 4 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: strict_capacity_limit : 0 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_compressed: (nil) 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: persistent_cache: (nil) 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_size: 4096 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_size_deviation: 10 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_restart_interval: 16 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_block_restart_interval: 1 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: metadata_block_size: 4096 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: partition_filters: 0 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: use_delta_encoding: 1 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: filter_policy: bloomfilter 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: whole_key_filtering: 1 2024-04-25T01:18:55.481 INFO:journalctl@ceph.mon.b.smithi038.stdout: verify_compression: 0 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: format_version: 5 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: enable_index_compression: 1 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_align: 0 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: max_auto_readahead_size: 262144 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: prepopulate_block_cache: 0 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression: NoCompression 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:18:55.482 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.num_levels: 7 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:18:55.483 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:18:55.484 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:18:55.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:18:55.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.ttl: 2592000 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:18:55.487 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 80c53b36-bb60-49f1-b128-5cf114c013d1 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007935226686, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007935227170, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714007935, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "80c53b36-bb60-49f1-b128-5cf114c013d1", "db_session_id": "R3LNU15GI7LAGWGXCDEJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714007935227252, "job": 1, "event": "recovery_finished"} 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cb7dc7e000 2024-04-25T01:18:55.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: DB pointer 0x55cb7dc68000 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** DB Stats ** 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:18:55.489 INFO:journalctl@ceph.mon.b.smithi038.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:18:55.490 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:18:55.491 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:18:55.491 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:18:55.491 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:18:55.491 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:18:55.491 INFO:journalctl@ceph.mon.b.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: Block cache BinnedLRUCache@0x55cb7db8d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: using public_addr v2:172.21.15.38:0/0 -> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: starting mon.b rank -1 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(???) e0 preinit fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: e1 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:18:55.492 INFO:journalctl@ceph.mon.b.smithi038.stdout: legacy client fscid: -1 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout: No filesystems configured 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-25T01:18:55.494 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Deploying cephadm binary to smithi156 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Added host smithi156 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.495 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1317401700' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:18:55.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1317401700' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c;count:3 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:18:55.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Deploying daemon mon.c on smithi156 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.a calling monitor election 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.c calling monitor election 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.498 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: monmap epoch 2 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: last_changed 2024-04-25T01:18:46.190029+0000 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: min_mon_release 19 (squid) 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: election_strategy: 1 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: fsmap 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mgrmap e13: x(active, since 51s) 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: overall HEALTH_OK 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.499 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: Deploying daemon mon.b on smithi038 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/1123978964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 ceph-mon[32841]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 bash[32822]: 97c40aa9d4810b99251505da6d09efd8a3c61619c42455529904d9f9931614be 2024-04-25T01:18:55.500 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:18:55 smithi038 systemd[1]: Started Ceph mon.b for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:19:00.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 -- 172.21.15.156:0/2356799785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 msgr2=0x7f1b0c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:00.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 --2- 172.21.15.156:0/2356799785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b0c005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1b2406a520 tx=0x7f1b1400fe60 comp rx=0 tx=0).stop 2024-04-25T01:19:00.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 -- 172.21.15.156:0/2356799785 shutdown_connections 2024-04-25T01:19:00.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 --2- 172.21.15.156:0/2356799785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b0c005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 -- 172.21.15.156:0/2356799785 >> 172.21.15.156:0/2356799785 conn(0x7f1b240fc770 msgr2=0x7f1b240feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:00.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 -- 172.21.15.156:0/2356799785 shutdown_connections 2024-04-25T01:19:00.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.408+0000 7f1b2a040700 1 -- 172.21.15.156:0/2356799785 wait complete. 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.409+0000 7f1b2a040700 1 Processor -- start 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.409+0000 7f1b2a040700 1 -- start start 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b2a040700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b2419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b2a040700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 0x7f1b24199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b2a040700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b2a040700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24102520 con 0x7f1b0c0035c0 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b2a040700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241023a0 con 0x7f1b24199f70 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b2a040700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241026a0 con 0x7f1b2419fdb0 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b2419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.156:57062/0 (socket says 172.21.15.156:57062) 2024-04-25T01:19:00.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 -- 172.21.15.156:0/3710002995 learned_addr learned my addr 172.21.15.156:0/3710002995 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b2419f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.156:44804/0 (socket says 172.21.15.156:44804) 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b22ffd700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 0x7f1b24199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 msgr2=0x7f1b2419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 msgr2=0x7f1b2419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b23fff700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b22ffd700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 msgr2=0x7f1b24199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b22ffd700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 msgr2=0x7f1b24199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b22ffd700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 0x7f1b24199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b22ffd700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 0x7f1b24199a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 msgr2=0x7f1b24199a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 0x7f1b24199a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 msgr2=0x7f1b2419a420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.410+0000 7f1b237fe700 1 -- 172.21.15.156:0/3710002995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b241a6740 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.411+0000 7f1b237fe700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b2419f870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1b2406a520 tx=0x7f1b140055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.411+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1402b020 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.411+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b1401d440 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.411+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b241a6910 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.411+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b14022040 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.411+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b241a6ea0 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.412+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1b1402e020 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.412+0000 7f1b20ff9700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1b10039580 0x7f1b1003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.412+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b24063890 con 0x7f1b0c0035c0 2024-04-25T01:19:00.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.412+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1b1401d960 con 0x7f1b0c0035c0 2024-04-25T01:19:00.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.412+0000 7f1b22ffd700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1b10039580 0x7f1b1003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:00.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.413+0000 7f1b22ffd700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1b10039580 0x7f1b1003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009850 tx=0x7f1b18008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:00.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.418+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b14035020 con 0x7f1b0c0035c0 2024-04-25T01:19:00.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:19:00.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: mon.a calling monitor election 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: mon.c calling monitor election 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: mon.b calling monitor election 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: monmap epoch 3 2024-04-25T01:19:00.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: last_changed 2024-04-25T01:18:55.252473+0000 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: min_mon_release 19 (squid) 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: election_strategy: 1 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: fsmap 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: mgrmap e13: x(active, since 60s) 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: overall HEALTH_OK 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:00.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:00.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.707+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1b241a70d0 con 0x7f1b0c0035c0 2024-04-25T01:19:00.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.708+0000 7f1b20ff9700 1 -- 172.21.15.156:0/3710002995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f1b140252d0 con 0x7f1b0c0035c0 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: mon.a calling monitor election 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: mon.c calling monitor election 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: mon.b calling monitor election 2024-04-25T01:19:00.692 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: monmap epoch 3 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: last_changed 2024-04-25T01:18:55.252473+0000 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: min_mon_release 19 (squid) 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: election_strategy: 1 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-25T01:19:00.693 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: fsmap 2024-04-25T01:19:00.694 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:19:00.694 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: mgrmap e13: x(active, since 60s) 2024-04-25T01:19:00.694 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: overall HEALTH_OK 2024-04-25T01:19:00.694 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:00.694 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:00.694 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T01:19:00.694 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","modified":"2024-04-25T01:18:55.252473Z","created":"2024-04-25T01:16:49.048092Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1b10039580 msgr2=0x7f1b1003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1b10039580 0x7f1b1003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009850 tx=0x7f1b18008040 comp rx=0 tx=0).stop 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 msgr2=0x7f1b2419f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b2419f870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1b2406a520 tx=0x7f1b140055a0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 shutdown_connections 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1b10039580 0x7f1b1003ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.695 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24199f70 0x7f1b2419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b0c0035c0 0x7f1b2419f870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 --2- 172.21.15.156:0/3710002995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b2419fdb0 0x7f1b24199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 >> 172.21.15.156:0/3710002995 conn(0x7f1b240fc770 msgr2=0x7f1b24103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:00.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 shutdown_connections 2024-04-25T01:19:00.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:00.711+0000 7f1b2a040700 1 -- 172.21.15.156:0/3710002995 wait complete. 2024-04-25T01:19:00.696 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 3 2024-04-25T01:19:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:19:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: mon.a calling monitor election 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: mon.c calling monitor election 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: mon.b calling monitor election 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: monmap epoch 3 2024-04-25T01:19:00.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: last_changed 2024-04-25T01:18:55.252473+0000 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: created 2024-04-25T01:16:49.048092+0000 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: min_mon_release 19 (squid) 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: election_strategy: 1 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: fsmap 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: mgrmap e13: x(active, since 60s) 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: overall HEALTH_OK 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:00.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:01 smithi156 ceph-mon[32631]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:01.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:01.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:01 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/3710002995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:19:01.415 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:01.445 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T01:19:01.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T01:19:01.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:01 smithi016 ceph-mon[29153]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:01.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:01.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:01 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/3710002995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:19:01.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:19:01.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:01 smithi038 ceph-mon[32841]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:01.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:01.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:01.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:01 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/3710002995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:19:01.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:19:02.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: Updating smithi016:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:02.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:02 smithi038 ceph-mon[32841]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:02.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:02.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:02.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: Updating smithi016:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:02.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:02 smithi016 ceph-mon[29153]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:03.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-25T01:19:03.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-25T01:19:03.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T01:19:03.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: Updating smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:03.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: Updating smithi016:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:03.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:02 smithi156 ceph-mon[32631]: Updating smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:19:03.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 -- 172.21.15.16:0/372271715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0101ea0 msgr2=0x7f68d0102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/372271715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0101ea0 0x7f68d0102280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009910 tx=0x7f68c0009c20 comp rx=0 tx=0).stop 2024-04-25T01:19:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 -- 172.21.15.16:0/372271715 shutdown_connections 2024-04-25T01:19:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/372271715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0101ea0 0x7f68d0102280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 -- 172.21.15.16:0/372271715 >> 172.21.15.16:0/372271715 conn(0x7f68d00fd750 msgr2=0x7f68d00ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 -- 172.21.15.16:0/372271715 shutdown_connections 2024-04-25T01:19:03.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.513+0000 7f68d5c2a700 1 -- 172.21.15.16:0/372271715 wait complete. 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.514+0000 7f68d5c2a700 1 Processor -- start 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.514+0000 7f68d5c2a700 1 -- start start 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68d5c2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68d0101ea0 0x7f68d019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68d5c2a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68d019fd50 0x7f68d01999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68d5c2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 0x7f68d019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68d5c2a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68d010c120 con 0x7f68d0199f10 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68d5c2a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f68d010bfa0 con 0x7f68d0101ea0 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68d5c2a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68d010c2a0 con 0x7f68d019fd50 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 0x7f68d019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 0x7f68d019a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43882/0 (socket says 172.21.15.16:43882) 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 -- 172.21.15.16:0/3008769035 learned_addr learned my addr 172.21.15.16:0/3008769035 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:19:03.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cf7fe700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68d0101ea0 0x7f68d019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68ceffd700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68d019fd50 0x7f68d01999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 -- 172.21.15.16:0/3008769035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68d019fd50 msgr2=0x7f68d01999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68d019fd50 0x7f68d01999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 -- 172.21.15.16:0/3008769035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68d0101ea0 msgr2=0x7f68d019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68d0101ea0 0x7f68d019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 -- 172.21.15.16:0/3008769035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d01a66e0 con 0x7f68d0199f10 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cf7fe700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68d0101ea0 0x7f68d019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68cffff700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 0x7f68d019a3c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68c800ba50 tx=0x7f68c800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.515+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c8019070 con 0x7f68d0199f10 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.516+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d01a6910 con 0x7f68d0199f10 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.516+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68c8014440 con 0x7f68d0199f10 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.516+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c800ce60 con 0x7f68d0199f10 2024-04-25T01:19:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.516+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d01a6e00 con 0x7f68d0199f10 2024-04-25T01:19:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.517+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f68c801c070 con 0x7f68d0199f10 2024-04-25T01:19:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.517+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d0063890 con 0x7f68d0199f10 2024-04-25T01:19:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.517+0000 7f68ccff9700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f68b8039580 0x7f68b803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.517+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f68c804b800 con 0x7f68d0199f10 2024-04-25T01:19:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.517+0000 7f68cf7fe700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f68b8039580 0x7f68b803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.517+0000 7f68cf7fe700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f68b8039580 0x7f68b803ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68c0005e00 tx=0x7f68c001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.522+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68c801b2c0 con 0x7f68d0199f10 2024-04-25T01:19:03.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.733+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f68d01a7090 con 0x7f68d0199f10 2024-04-25T01:19:03.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.734+0000 7f68ccff9700 1 -- 172.21.15.16:0/3008769035 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f68c8018170 con 0x7f68d0199f10 2024-04-25T01:19:03.718 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:19:03.718 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-04-25T01:19:03.718 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:19:03.718 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f68b8039580 msgr2=0x7f68b803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f68b8039580 0x7f68b803ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68c0005e00 tx=0x7f68c001a040 comp rx=0 tx=0).stop 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 msgr2=0x7f68d019a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 0x7f68d019a3c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68c800ba50 tx=0x7f68c800bd60 comp rx=0 tx=0).stop 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 shutdown_connections 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f68b8039580 0x7f68b803ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d0199f10 0x7f68d019a3c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68d0101ea0 0x7f68d019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 --2- 172.21.15.16:0/3008769035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68d019fd50 0x7f68d01999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 >> 172.21.15.16:0/3008769035 conn(0x7f68d00fd750 msgr2=0x7f68d00745a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 shutdown_connections 2024-04-25T01:19:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:03.736+0000 7f68d5c2a700 1 -- 172.21.15.16:0/3008769035 wait complete. 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: Reconfiguring daemon mon.a on smithi016 2024-04-25T01:19:04.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:03 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3008769035' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:04.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: Reconfiguring daemon mon.a on smithi016 2024-04-25T01:19:04.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:03 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3008769035' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:04.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:19:04.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:04.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:04.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:04.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: Reconfiguring daemon mon.a on smithi016 2024-04-25T01:19:04.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:03 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3008769035' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:04.471 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T01:19:04.472 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:19:04.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:19:04.502 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:19:04.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:19:04.571 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:19:04.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:19:04.604 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:19:04.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:19:04.672 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:19:04.672 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:19:04.704 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:19:04.704 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:19:04.772 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-04-25T01:19:04.772 INFO:tasks.cephadm:Adding mgr.y on smithi038 2024-04-25T01:19:04.772 INFO:tasks.cephadm:Adding mgr.z on smithi156 2024-04-25T01:19:04.772 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi016=x;smithi038=y;smithi156=z' 2024-04-25T01:19:05.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:04 smithi156 ceph-mon[32631]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:05.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:05.045 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:19:05.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:04 smithi016 ceph-mon[29153]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:05.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:05.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:04 smithi038 ceph-mon[32841]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:05.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 -- 172.21.15.156:0/769471212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c101db0 msgr2=0x7fd20c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 --2- 172.21.15.156:0/769471212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c101db0 0x7fd20c1021d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd208009910 tx=0x7fd208009c20 comp rx=0 tx=0).stop 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 -- 172.21.15.156:0/769471212 shutdown_connections 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 --2- 172.21.15.156:0/769471212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c101db0 0x7fd20c1021d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 -- 172.21.15.156:0/769471212 >> 172.21.15.156:0/769471212 conn(0x7fd20c0fd350 msgr2=0x7fd20c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 -- 172.21.15.156:0/769471212 shutdown_connections 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.472+0000 7fd213a7a700 1 -- 172.21.15.156:0/769471212 wait complete. 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 Processor -- start 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 -- start start 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20c101db0 0x7fd20c197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 0x7fd20c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c197ed0 0x7fd20c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c107cf0 con 0x7fd20c197ed0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c107b70 con 0x7fd20c101db0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.473+0000 7fd213a7a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c107e70 con 0x7fd20c197a50 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 0x7fd20c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211816700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20c101db0 0x7fd20c197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 0x7fd20c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:47224/0 (socket says 172.21.15.156:47224) 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 -- 172.21.15.156:0/3236278263 learned_addr learned my addr 172.21.15.156:0/3236278263 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd212017700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c197ed0 0x7fd20c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 -- 172.21.15.156:0/3236278263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20c101db0 msgr2=0x7fd20c197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20c101db0 0x7fd20c197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 -- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c197ed0 msgr2=0x7fd20c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c197ed0 0x7fd20c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 -- 172.21.15.156:0/3236278263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c19f920 con 0x7fd20c197a50 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211816700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20c101db0 0x7fd20c197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:19:06.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd212017700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c197ed0 0x7fd20c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd211015700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 0x7fd20c19cac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b780 tx=0x7fd1fc00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc016070 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1fc0036a0 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0119c0 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.474+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c0744c0 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.475+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c074ad0 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.475+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd1fc019070 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.475+0000 7fd202ffd700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1f8039580 0x7fd1f803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.475+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c063890 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.476+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd1fc04ad10 con 0x7fd20c197a50 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.476+0000 7fd211816700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1f8039580 0x7fd1f803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.476+0000 7fd211816700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1f8039580 0x7fd1f803ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd208005190 tx=0x7fd20801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.464 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.481+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1fc019970 con 0x7fd20c197a50 2024-04-25T01:19:06.670 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.686+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi038=y;smithi156=z", "target": ["mon-mgr", ""]}) -- 0x7fd20c0627c0 con 0x7fd1f8039580 2024-04-25T01:19:06.670 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.670 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:06 smithi156 ceph-mon[32631]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:19:06.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:06.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:06.671 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.693+0000 7fd202ffd700 1 -- 172.21.15.156:0/3236278263 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd20c0627c0 con 0x7fd1f8039580 2024-04-25T01:19:06.677 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mgr update... 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1f8039580 msgr2=0x7fd1f803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1f8039580 0x7fd1f803ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd208005190 tx=0x7fd20801a040 comp rx=0 tx=0).stop 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 msgr2=0x7fd20c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 0x7fd20c19cac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b780 tx=0x7fd1fc00ba90 comp rx=0 tx=0).stop 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 shutdown_connections 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1f8039580 0x7fd1f803ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20c197ed0 0x7fd20c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20c101db0 0x7fd20c197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 --2- 172.21.15.156:0/3236278263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd20c197a50 0x7fd20c19cac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 >> 172.21.15.156:0/3236278263 conn(0x7fd20c0fd350 msgr2=0x7fd20c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 shutdown_connections 2024-04-25T01:19:06.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:19:06.695+0000 7fd213a7a700 1 -- 172.21.15.156:0/3236278263 wait complete. 2024-04-25T01:19:06.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:06 smithi038 ceph-mon[32841]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:19:06.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:06.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:06.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:06.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:06.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:06 smithi016 ceph-mon[29153]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:19:06.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:06.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:06.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:07.211 DEBUG:teuthology.orchestra.run.smithi038:mgr.y> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T01:19:07.214 DEBUG:teuthology.orchestra.run.smithi156:mgr.z> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T01:19:07.216 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T01:19:07.216 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:19:07.216 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:19:07.236 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:19:07.236 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-25 01:18:13.380758956 +0000 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-25 01:13:28.482754198 +0000 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-25 01:13:28.482754198 +0000 2024-04-25T01:19:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-25 01:13:28.482754198 +0000 2024-04-25T01:19:07.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:19:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-25T01:19:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-25T01:19:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000121448 s, 4.2 MB/s 2024-04-25T01:19:07.363 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:19:07.422 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-25 01:18:13.411758475 +0000 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-25 01:13:28.819748958 +0000 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-25 01:13:28.819748958 +0000 2024-04-25T01:19:07.481 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-25 01:13:28.819748958 +0000 2024-04-25T01:19:07.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:19:07.546 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-25T01:19:07.546 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-25T01:19:07.546 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000122479 s, 4.2 MB/s 2024-04-25T01:19:07.547 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:19:07.605 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-25T01:19:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: Reconfiguring daemon mon.b on smithi038 2024-04-25T01:19:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:07.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:07.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:07 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:19:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:07.629 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T01:19:07.629 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:07.629 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:07.629 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-25 01:18:13.440758024 +0000 2024-04-25T01:19:07.630 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-25 01:13:29.164743593 +0000 2024-04-25T01:19:07.630 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-25 01:13:29.164743593 +0000 2024-04-25T01:19:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-25 01:13:29.164743593 +0000 2024-04-25T01:19:07.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:19:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-25T01:19:07.694 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-25T01:19:07.695 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000114081 s, 4.5 MB/s 2024-04-25T01:19:07.696 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:19:07.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: Reconfiguring daemon mon.b on smithi038 2024-04-25T01:19:07.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:07.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:07.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:07.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:07 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:07.752 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-25T01:19:07.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: Reconfiguring daemon mon.b on smithi038 2024-04-25T01:19:07.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:07.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:07.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:19:07.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:19:07.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:07 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:07.809 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:19:07.809 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:07.809 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-25T01:19:07.809 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:07.809 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:07.809 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-25 01:18:13.469757573 +0000 2024-04-25T01:19:07.810 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-25 01:13:29.510738213 +0000 2024-04-25T01:19:07.810 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-25 01:13:29.510738213 +0000 2024-04-25T01:19:07.810 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-25 01:13:29.510738213 +0000 2024-04-25T01:19:07.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:19:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-25T01:19:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-25T01:19:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000111471 s, 4.6 MB/s 2024-04-25T01:19:07.875 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:19:07.933 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-25T01:19:07.933 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:19:07.952 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:19:07.953 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-25T01:19:08.011 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:19:08.011 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T01:19:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-25 01:18:54.711917943 +0000 2024-04-25T01:19:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-25 01:13:30.236907582 +0000 2024-04-25T01:19:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-25 01:13:30.236907582 +0000 2024-04-25T01:19:08.012 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-25 01:13:30.236907582 +0000 2024-04-25T01:19:08.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:19:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-25T01:19:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-25T01:19:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000385844 s, 1.3 MB/s 2024-04-25T01:19:08.081 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:19:08.142 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-25 01:18:54.712917924 +0000 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-25 01:13:30.616900545 +0000 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-25 01:13:30.616900545 +0000 2024-04-25T01:19:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-25 01:13:30.616900545 +0000 2024-04-25T01:19:08.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:19:08.271 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-25T01:19:08.271 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-25T01:19:08.271 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00217165 s, 236 kB/s 2024-04-25T01:19:08.273 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:19:08.335 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-25T01:19:08.394 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:19:08.394 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:08.394 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T01:19:08.394 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:08.394 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:08.394 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-25 01:18:54.712917924 +0000 2024-04-25T01:19:08.395 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-25 01:13:30.987893674 +0000 2024-04-25T01:19:08.395 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-25 01:13:30.987893674 +0000 2024-04-25T01:19:08.395 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-25 01:13:30.987893674 +0000 2024-04-25T01:19:08.395 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:19:08.463 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-25T01:19:08.463 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-25T01:19:08.463 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000441218 s, 1.2 MB/s 2024-04-25T01:19:08.465 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:19:08.524 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-25T01:19:08.582 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:19:08.582 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:08.582 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-25T01:19:08.582 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:08.582 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:08.582 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-25 01:18:54.713917906 +0000 2024-04-25T01:19:08.583 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-25 01:13:31.341887117 +0000 2024-04-25T01:19:08.583 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-25 01:13:31.341887117 +0000 2024-04-25T01:19:08.583 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-25 01:13:31.341887117 +0000 2024-04-25T01:19:08.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:19:08.649 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-25T01:19:08.649 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-25T01:19:08.649 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000318309 s, 1.6 MB/s 2024-04-25T01:19:08.650 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:19:08.709 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T01:19:08.709 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi038=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: Saving service mgr spec with placement smithi016=x;smithi038=y;smithi156=z;count:3 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: Reconfiguring daemon mon.c on smithi156 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:08.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:08.726 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:19:08.726 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-04-25T01:19:08.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi038=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: Saving service mgr spec with placement smithi016=x;smithi038=y;smithi156=z;count:3 2024-04-25T01:19:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:19:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: Reconfiguring daemon mon.c on smithi156 2024-04-25T01:19:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 01:18:45.432111834 +0000 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 01:13:25.279097052 +0000 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 01:13:25.279097052 +0000 2024-04-25T01:19:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 01:13:25.279097052 +0000 2024-04-25T01:19:08.770 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:19:08.833 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T01:19:08.833 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T01:19:08.833 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.9882e-05 s, 5.1 MB/s 2024-04-25T01:19:08.834 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi038=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: Saving service mgr spec with placement smithi016=x;smithi038=y;smithi156=z;count:3 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: Reconfiguring daemon mon.c on smithi156 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:08.890 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 01:18:45.433111819 +0000 2024-04-25T01:19:08.946 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 01:13:25.618092038 +0000 2024-04-25T01:19:08.947 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 01:13:25.618092038 +0000 2024-04-25T01:19:08.947 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 01:13:25.618092038 +0000 2024-04-25T01:19:08.947 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:19:09.009 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T01:19:09.009 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T01:19:09.009 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.5624e-05 s, 5.4 MB/s 2024-04-25T01:19:09.010 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:19:09.066 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 01:18:45.433111819 +0000 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 01:13:25.942087245 +0000 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 01:13:25.942087245 +0000 2024-04-25T01:19:09.123 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 01:13:25.942087245 +0000 2024-04-25T01:19:09.124 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:19:09.186 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T01:19:09.186 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T01:19:09.186 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.7761e-05 s, 5.8 MB/s 2024-04-25T01:19:09.187 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:19:09.244 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 01:18:45.433111819 +0000 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 01:13:26.261082526 +0000 2024-04-25T01:19:09.300 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 01:13:26.261082526 +0000 2024-04-25T01:19:09.301 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 01:13:26.261082526 +0000 2024-04-25T01:19:09.301 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:19:09.364 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T01:19:09.364 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T01:19:09.364 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.6768e-05 s, 5.3 MB/s 2024-04-25T01:19:09.366 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:19:09.423 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-04-25T01:19:09.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:19:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:09.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:09.734 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:09 smithi156 ceph-mon[32631]: Deploying daemon mgr.z on smithi156 2024-04-25T01:19:09.794 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:09.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:09 smithi016 ceph-mon[29153]: Deploying daemon mgr.z on smithi156 2024-04-25T01:19:09.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:09.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:09.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:09.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:09.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:09.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:19:09.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:09.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:09.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:09 smithi038 ceph-mon[32841]: Deploying daemon mgr.z on smithi156 2024-04-25T01:19:10.648 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:10 smithi156 ceph-mon[32631]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:10.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:10 smithi016 ceph-mon[29153]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:10.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:10 smithi038 ceph-mon[32841]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:11.004 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:10 smithi156 systemd[1]: Starting Ceph mgr.z for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:19:11.325 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 podman[34485]: 2024-04-25 01:19:11.048427218 +0000 UTC m=+0.113123971 container create ff88161473000fcaee15ca0aab27961d8b2d23b3b6394015336b60fe622b3644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T01:19:11.325 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 podman[34485]: 2024-04-25 01:19:10.953141398 +0000 UTC m=+0.017838127 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:19:11.325 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 podman[34485]: 2024-04-25 01:19:11.259306812 +0000 UTC m=+0.324003522 container init ff88161473000fcaee15ca0aab27961d8b2d23b3b6394015336b60fe622b3644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T01:19:11.325 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 podman[34485]: 2024-04-25 01:19:11.261783396 +0000 UTC m=+0.326480113 container start ff88161473000fcaee15ca0aab27961d8b2d23b3b6394015336b60fe622b3644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:19:11.325 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 ceph-mgr[34504]: -- 172.21.15.156:0/3098407701 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627d8da65a0 con 0x5627d8d7ac00 2024-04-25T01:19:11.325 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 bash[34485]: ff88161473000fcaee15ca0aab27961d8b2d23b3b6394015336b60fe622b3644 2024-04-25T01:19:11.581 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 systemd[1]: Started Ceph mgr.z for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:19:11.581 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:11.414+0000 7fc3ad5db200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:19:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:19:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:19:11.607 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-25T01:19:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-25T01:19:11.607 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00828986 s, 1.3 GB/s 2024-04-25T01:19:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-25T01:19:11.994 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:11.597+0000 7fc3ad5db200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:19:11.994 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:11.700+0000 7fc3ad5db200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:19:11.994 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:12.011+0000 7fc3ad5db200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:19:12.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-04-25T01:19:12.253 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:12.095+0000 7fc3ad5db200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:19:12.450 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:12.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:12 smithi016 ceph-mon[29153]: Deploying daemon mgr.y on smithi038 2024-04-25T01:19:12.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:12.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:19:12.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:12.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:12.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:12 smithi038 ceph-mon[32841]: Deploying daemon mgr.y on smithi038 2024-04-25T01:19:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:12.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:12 smithi156 ceph-mon[32631]: Deploying daemon mgr.y on smithi038 2024-04-25T01:19:13.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:13 smithi038 ceph-mon[32841]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:13.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:13 smithi016 ceph-mon[29153]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:13.636 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:13 smithi156 ceph-mon[32631]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:13.636 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:13.309+0000 7fc3ad5db200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:19:13.636 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:13.481+0000 7fc3ad5db200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:19:13.636 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:13.652+0000 7fc3ad5db200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:19:13.855 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:13 smithi038 ceph-mgr[34217]: -- 172.21.15.38:0/3396894428 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5565d77765a0 con 0x5565d774b000 2024-04-25T01:19:13.855 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:13 smithi038 bash[34197]: 68bae53df6defa01d510382883862228a561c0f89e49aac85cd4308252241afe 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 -- 172.21.15.16:0/3016207020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3768102d50 msgr2=0x7f3768109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 --2- 172.21.15.16:0/3016207020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3768102d50 0x7f3768109600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f376000b0d0 tx=0x7f376000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 -- 172.21.15.16:0/3016207020 shutdown_connections 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 --2- 172.21.15.16:0/3016207020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3768102d50 0x7f3768109600 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 --2- 172.21.15.16:0/3016207020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 --2- 172.21.15.16:0/3016207020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37681023b0 0x7f3768102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.138+0000 7f376f52f700 1 -- 172.21.15.16:0/3016207020 >> 172.21.15.16:0/3016207020 conn(0x7f37680fc750 msgr2=0x7f37680feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.139+0000 7f376f52f700 1 -- 172.21.15.16:0/3016207020 shutdown_connections 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.139+0000 7f376f52f700 1 -- 172.21.15.16:0/3016207020 wait complete. 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.139+0000 7f376f52f700 1 Processor -- start 2024-04-25T01:19:14.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.139+0000 7f376f52f700 1 -- start start 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376f52f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376f52f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37681023b0 0x7f3768197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376f52f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768102d50 0x7f376819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376f52f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f376810cd00 con 0x7f37681023b0 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376f52f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f376810cb80 con 0x7f37681011b0 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376f52f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f376810ce80 con 0x7f3768102d50 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37681023b0 0x7f3768197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376d2cb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376d2cb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:45042/0 (socket says 172.21.15.16:45042) 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376d2cb700 1 -- 172.21.15.16:0/282608300 learned_addr learned my addr 172.21.15.16:0/282608300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376dacc700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768102d50 0x7f376819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 -- 172.21.15.16:0/282608300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768102d50 msgr2=0x7f376819e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768102d50 0x7f376819e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:14.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 -- 172.21.15.16:0/282608300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 msgr2=0x7f37681972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 -- 172.21.15.16:0/282608300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376819ef40 con 0x7f37681023b0 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376d2cb700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f376caca700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37681023b0 0x7f3768197820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f375800b480 tx=0x7f375800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.140+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758015020 con 0x7f37681023b0 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.141+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f375800be50 con 0x7f37681023b0 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.141+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376819f230 con 0x7f37681023b0 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.141+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758010bd0 con 0x7f37681023b0 2024-04-25T01:19:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.141+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376819f7a0 con 0x7f37681023b0 2024-04-25T01:19:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.141+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3758018020 con 0x7f37681023b0 2024-04-25T01:19:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.141+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3768068f20 con 0x7f37681023b0 2024-04-25T01:19:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.142+0000 7f375e7fc700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3754039330 0x7f375403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.142+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f375804ba50 con 0x7f37681023b0 2024-04-25T01:19:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.142+0000 7f376d2cb700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3754039330 0x7f375403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:14.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.142+0000 7f376d2cb700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3754039330 0x7f375403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37640105c0 tx=0x7f376400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:14.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.147+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f375801f020 con 0x7f37681023b0 2024-04-25T01:19:14.138 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:13 smithi038 systemd[1]: Started Ceph mgr.y for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:19:14.139 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:13 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:13.959+0000 7f0011105200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:19:14.139 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:14.156+0000 7f0011105200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:19:14.227 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:13.939+0000 7fc3ad5db200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:19:14.227 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:14.033+0000 7fc3ad5db200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:19:14.227 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:14.244+0000 7fc3ad5db200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:19:14.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:14.360+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3768062670 con 0x7f3754039330 2024-04-25T01:19:14.459 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:14.268+0000 7f0011105200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:19:14.504 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:14.445+0000 7fc3ad5db200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:19:14.719 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:14.613+0000 7f0011105200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:19:14.719 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:14.712+0000 7f0011105200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:19:15.183 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:15.183 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:15.184 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:15.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:15.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:15.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:15.220 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:15 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:15.095+0000 7fc3ad5db200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:15.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:15.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:15.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:15.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:15.813 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:15 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:15.530+0000 7fc3ad5db200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:19:15.813 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:15 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:15.628+0000 7fc3ad5db200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:19:15.813 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:15 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:15.829+0000 7fc3ad5db200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:19:16.383 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:16.048+0000 7f0011105200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:19:16.383 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:16.225+0000 7f0011105200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:19:16.384 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:16.400+0000 7f0011105200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:19:16.678 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:16.358+0000 7fc3ad5db200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:19:16.678 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:16.448+0000 7fc3ad5db200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:19:16.678 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.678 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:16.679 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:16.683 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.683 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:16.684 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:16.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:16.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:16.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:19:16.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:19:16.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:16.969 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:16.701+0000 7f0011105200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:19:16.969 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:16.796+0000 7f0011105200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:19:17.003 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:16.694+0000 7fc3ad5db200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:19:17.003 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:16.826+0000 7fc3ad5db200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:19:17.003 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:16.922+0000 7fc3ad5db200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:19:17.455 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:17.039+0000 7fc3ad5db200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:19:17.455 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:17.131+0000 7fc3ad5db200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:19:17.456 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:17.005+0000 7f0011105200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:19:17.456 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:17.204+0000 7f0011105200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: Reconfiguring daemon mgr.x on smithi016 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1111141666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b40650a-6f71-49c8-b04f-f76ed7b91b72"}]: dispatch 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1111141666' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b40650a-6f71-49c8-b04f-f76ed7b91b72"}]': finished 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:19:17.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:17.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:17.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:19:17.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: Reconfiguring daemon mgr.x on smithi016 2024-04-25T01:19:17.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1111141666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b40650a-6f71-49c8-b04f-f76ed7b91b72"}]: dispatch 2024-04-25T01:19:17.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1111141666' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b40650a-6f71-49c8-b04f-f76ed7b91b72"}]': finished 2024-04-25T01:19:17.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:19:17.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:17.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:17.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:19:17.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: Reconfiguring daemon mgr.x on smithi016 2024-04-25T01:19:17.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1111141666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b40650a-6f71-49c8-b04f-f76ed7b91b72"}]: dispatch 2024-04-25T01:19:17.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1111141666' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b40650a-6f71-49c8-b04f-f76ed7b91b72"}]': finished 2024-04-25T01:19:17.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:19:17.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:17 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:18.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:17.842+0000 7f0011105200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:19:18.253 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:17 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:17.952+0000 7fc3ad5db200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:19:18.253 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z[34500]: 2024-04-25T01:19:18.165+0000 7fc3ad5db200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:19:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:18.482+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f37580193b0 con 0x7f37681023b0 2024-04-25T01:19:18.574 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:18.282+0000 7f0011105200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:19:18.574 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:18.382+0000 7f0011105200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:18.591+0000 7f0011105200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2523668447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-mon[32841]: Standby manager daemon z started 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:19:18.575 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:18 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:19:18.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2523668447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:18.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-mon[32631]: Standby manager daemon z started 2024-04-25T01:19:18.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:19:18.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:19:18.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:19:18.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:18 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:19:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:18 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2523668447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:18 smithi016 ceph-mon[29153]: Standby manager daemon z started 2024-04-25T01:19:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:18 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:19:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:18 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:19:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:18 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:19:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:18 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.156:0/1919890392' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:19:19.462 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.141+0000 7f0011105200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:19:19.462 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.235+0000 7f0011105200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:19:19.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-mon[32841]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:19.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-mon[32841]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:19:19.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:19:19.719 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.487+0000 7f0011105200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:19:19.719 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.625+0000 7f0011105200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:19:19.719 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.723+0000 7f0011105200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:19:19.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:19 smithi156 ceph-mon[32631]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:19.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:19 smithi156 ceph-mon[32631]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:19:19.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:19:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:19 smithi016 ceph-mon[29153]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:19 smithi016 ceph-mon[29153]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:19:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:19:20.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.843+0000 7f0011105200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:19:20.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:19.939+0000 7f0011105200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:19:20.779 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:20 smithi038 ceph-mon[32841]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:20.780 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:20 smithi016 ceph-mon[29153]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:21.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:20 smithi156 ceph-mon[32631]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:21.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:20 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:20.797+0000 7f0011105200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:19:21.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:19:21 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y[34213]: 2024-04-25T01:19:21.019+0000 7f0011105200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:19:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:21.545+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f375805e960 con 0x7f37681023b0 2024-04-25T01:19:21.681 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:21 smithi016 ceph-mon[29153]: Standby manager daemon y started 2024-04-25T01:19:21.681 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:21 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:19:21.681 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:21 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:19:21.681 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:21 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:19:21.681 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:21 smithi016 ceph-mon[29153]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:19:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:21 smithi038 ceph-mon[32841]: Standby manager daemon y started 2024-04-25T01:19:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:21 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:19:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:21 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:19:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:21 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:19:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:21 smithi038 ceph-mon[32841]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:19:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:21 smithi156 ceph-mon[32631]: Standby manager daemon y started 2024-04-25T01:19:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:21 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:19:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:21 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:19:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:21 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:19:22.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:21 smithi156 ceph-mon[32631]: from='mgr.? 172.21.15.38:0/3525669814' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:19:22.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:22 smithi016 ceph-mon[29153]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:22.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:22 smithi016 ceph-mon[29153]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T01:19:22.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:19:22.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:22 smithi156 ceph-mon[32631]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:22.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:22 smithi156 ceph-mon[32631]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T01:19:22.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:19:22.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:22 smithi038 ceph-mon[32841]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:22.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:22 smithi038 ceph-mon[32841]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T01:19:22.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:19:23.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:23.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:23.750 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:24.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:24.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:24.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:24.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:24.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:24.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:25.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:24 smithi156 ceph-mon[32631]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:25.022 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:24 smithi016 ceph-mon[29153]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:25.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:24 smithi038 ceph-mon[32841]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:26.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:25 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:19:26.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:25 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:26.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:25 smithi156 ceph-mon[32631]: Deploying daemon osd.0 on smithi016 2024-04-25T01:19:26.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:25 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:19:26.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:25 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:26.016 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:25 smithi016 ceph-mon[29153]: Deploying daemon osd.0 on smithi016 2024-04-25T01:19:26.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:25 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:19:26.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:25 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:26.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:25 smithi038 ceph-mon[32841]: Deploying daemon osd.0 on smithi016 2024-04-25T01:19:26.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:26 smithi016 ceph-mon[29153]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:26 smithi156 ceph-mon[32631]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:27.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:26 smithi038 ceph-mon[32841]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:28.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:28 smithi016 ceph-mon[29153]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:29.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:28 smithi156 ceph-mon[32631]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:29.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:28 smithi038 ceph-mon[32841]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:30.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:30 smithi016 ceph-mon[29153]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:31.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:30 smithi156 ceph-mon[32631]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:31.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:30 smithi038 ceph-mon[32841]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:32.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:32.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:32.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:32.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:33.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:32 smithi016 ceph-mon[29153]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:33.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:32 smithi038 ceph-mon[32841]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:33.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:32 smithi156 ceph-mon[32631]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:34.762 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:34.763 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:34.763 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:34.763 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:34.763 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:34.763 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:34 smithi016 ceph-mon[29153]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:34.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.838+0000 7f375e7fc700 1 -- 172.21.15.16:0/282608300 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3768062670 con 0x7f3754039330 2024-04-25T01:19:34.821 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3754039330 msgr2=0x7f375403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3754039330 0x7f375403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37640105c0 tx=0x7f376400b4a0 comp rx=0 tx=0).stop 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37681023b0 msgr2=0x7f3768197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37681023b0 0x7f3768197820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f375800b480 tx=0x7f375800b790 comp rx=0 tx=0).stop 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 shutdown_connections 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3754039330 0x7f375403b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768102d50 0x7f376819e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37681011b0 0x7f37681972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 --2- 172.21.15.16:0/282608300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37681023b0 0x7f3768197820 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 >> 172.21.15.16:0/282608300 conn(0x7f37680fc750 msgr2=0x7f3768105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 shutdown_connections 2024-04-25T01:19:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:19:34.840+0000 7f376f52f700 1 -- 172.21.15.16:0/282608300 wait complete. 2024-04-25T01:19:34.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:34.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:34.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:34.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:34.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:34.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:34 smithi038 ceph-mon[32841]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:35.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:35.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:35.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:34 smithi156 ceph-mon[32631]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:35.547 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T01:19:35.549 INFO:tasks.cephadm:Deploying osd.1 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-25T01:19:35.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:19:35.803 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:19:35.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:35 smithi016 ceph-mon[29153]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:19:35.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:35 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:35 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.804 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:35 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:35.926 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:35 smithi038 ceph-mon[32841]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:19:35.927 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:35 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.927 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:35 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:35.927 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:35 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:36.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:35 smithi156 ceph-mon[32631]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:19:36.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:35 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:36.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:35 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:36.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:35 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:36.813 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:36 smithi016 ceph-mon[29153]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:36.813 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:36 smithi016 ceph-mon[29153]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:19:36.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:36 smithi016 ceph-mon[29153]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:19:36.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:36 smithi016 ceph-mon[29153]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:19:36.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:36.839 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:36 smithi038 ceph-mon[32841]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:36.840 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:36 smithi038 ceph-mon[32841]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:19:36.840 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:36 smithi038 ceph-mon[32841]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:19:36.840 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:36 smithi038 ceph-mon[32841]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:19:36.840 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:36 smithi156 ceph-mon[32631]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:37.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:36 smithi156 ceph-mon[32631]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:19:37.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:36 smithi156 ceph-mon[32631]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:19:37.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:36 smithi156 ceph-mon[32631]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:19:37.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.093 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0[43621]: 2024-04-25T01:19:37.031+0000 7fcbb38b9700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:19:37.093 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0[43621]: 2024-04-25T01:19:37.034+0000 7fcbab6a4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:19:37.683 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:19:37.684 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:19:37.684 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-25T01:19:37.684 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-25T01:19:37.684 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150641 s, 696 MB/s 2024-04-25T01:19:37.684 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-25T01:19:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-mon[29153]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:19:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-mon[29153]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:19:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-mon[29153]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' 2024-04-25T01:19:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.957 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:37 smithi038 ceph-mon[32841]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:19:37.957 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:37 smithi038 ceph-mon[32841]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:19:37.957 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.957 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:37.957 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:37 smithi038 ceph-mon[32841]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' 2024-04-25T01:19:37.957 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:38.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:37 smithi156 ceph-mon[32631]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:19:38.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:37 smithi156 ceph-mon[32631]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:19:38.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:38.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:38.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:37 smithi156 ceph-mon[32631]: from='osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960]' entity='osd.0' 2024-04-25T01:19:38.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:38.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-25T01:19:38.453 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:19:38.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:38 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:19:38.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:38 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:19:38.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:38 smithi038 ceph-mon[32841]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:38.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:38 smithi038 ceph-mon[32841]: osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] boot 2024-04-25T01:19:38.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:38 smithi038 ceph-mon[32841]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:19:38.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:38 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:38.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:38 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:19:38.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:38 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:19:38.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:38 smithi016 ceph-mon[29153]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:38.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:38 smithi016 ceph-mon[29153]: osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] boot 2024-04-25T01:19:38.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:38 smithi016 ceph-mon[29153]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:19:38.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:38 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:39.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:38 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:19:39.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:38 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:19:39.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:38 smithi156 ceph-mon[32631]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:19:39.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:38 smithi156 ceph-mon[32631]: osd.0 [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] boot 2024-04-25T01:19:39.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:38 smithi156 ceph-mon[32631]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:19:39.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:38 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi016 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi016 to 17093M 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:39.819 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi016 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi016 to 17093M 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:39.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi016 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi016 to 17093M 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 -- 172.21.15.38:0/505589989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41011d0 msgr2=0x7febb41015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 --2- 172.21.15.38:0/505589989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41011d0 0x7febb41015f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feba40099c0 tx=0x7feba4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 -- 172.21.15.38:0/505589989 shutdown_connections 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 --2- 172.21.15.38:0/505589989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb4102d70 0x7febb4109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 --2- 172.21.15.38:0/505589989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41011d0 0x7febb41015f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 --2- 172.21.15.38:0/505589989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febb41023d0 0x7febb4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 -- 172.21.15.38:0/505589989 >> 172.21.15.38:0/505589989 conn(0x7febb40fc750 msgr2=0x7febb40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 -- 172.21.15.38:0/505589989 shutdown_connections 2024-04-25T01:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.032+0000 7febb8d17700 1 -- 172.21.15.38:0/505589989 wait complete. 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 Processor -- start 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 -- start start 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb41011d0 0x7febb4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 0x7febb4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febb4102d70 0x7febb419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7febb410ccd0 con 0x7febb41011d0 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7febb410cb50 con 0x7febb41023d0 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb8d17700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7febb410ce50 con 0x7febb4102d70 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb1d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 0x7febb4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb1d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 0x7febb4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46364/0 (socket says 172.21.15.38:46364) 2024-04-25T01:19:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb1d9b700 1 -- 172.21.15.38:0/1629728980 learned_addr learned my addr 172.21.15.38:0/1629728980 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.033+0000 7febb2d9d700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febb4102d70 0x7febb419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb259c700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb41011d0 0x7febb4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb1d9b700 1 -- 172.21.15.38:0/1629728980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febb4102d70 msgr2=0x7febb419f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb1d9b700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febb4102d70 0x7febb419f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb1d9b700 1 -- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb41011d0 msgr2=0x7febb4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb1d9b700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb41011d0 0x7febb4198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb1d9b700 1 -- 172.21.15.38:0/1629728980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb41035f0 con 0x7febb41023d0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb1d9b700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 0x7febb4198970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feba800c970 tx=0x7feba800cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8007aa0 con 0x7febb41023d0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb41038e0 con 0x7febb41023d0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba8004ce0 con 0x7febb41023d0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba80055b0 con 0x7febb41023d0 2024-04-25T01:19:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.034+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb4103dd0 con 0x7febb41023d0 2024-04-25T01:19:40.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.035+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb4068f20 con 0x7febb41023d0 2024-04-25T01:19:40.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.036+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7feba8004750 con 0x7febb41023d0 2024-04-25T01:19:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.037+0000 7feba37fe700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb9c0a2b90 0x7feb9c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.037+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7feba80cd720 con 0x7febb41023d0 2024-04-25T01:19:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.037+0000 7febb259c700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb9c0a2b90 0x7feb9c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.038+0000 7febb259c700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb9c0a2b90 0x7feb9c0a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feba40038f0 tx=0x7feba4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:40.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.040+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feba809c180 con 0x7febb41023d0 2024-04-25T01:19:40.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:40.260+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7febb40fe8f0 con 0x7feb9c0a2b90 2024-04-25T01:19:40.880 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:40 smithi038 ceph-mon[32841]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:40.880 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:40 smithi038 ceph-mon[32841]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:19:40.880 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:40 smithi038 ceph-mon[32841]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:40.880 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:40.880 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:40.881 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:40.922 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:40 smithi156 ceph-mon[32631]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:40.922 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:40 smithi156 ceph-mon[32631]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:19:40.923 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:40 smithi156 ceph-mon[32631]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:40.923 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:40.923 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:40.923 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:40 smithi016 ceph-mon[29153]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:40 smithi016 ceph-mon[29153]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:19:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:40 smithi016 ceph-mon[29153]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:19:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:19:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:19:41.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:42.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:42 smithi038 ceph-mon[32841]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:43.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:42 smithi156 ceph-mon[32631]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:43.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:42 smithi016 ceph-mon[29153]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:43.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:43 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/387973365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]: dispatch 2024-04-25T01:19:43.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:43 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]: dispatch 2024-04-25T01:19:43.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:43 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]': finished 2024-04-25T01:19:43.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:43 smithi038 ceph-mon[32841]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:19:43.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:19:43.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:43 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/1233225932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:44.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:43 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/387973365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]: dispatch 2024-04-25T01:19:44.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:43 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]: dispatch 2024-04-25T01:19:44.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:43 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]': finished 2024-04-25T01:19:44.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:43 smithi156 ceph-mon[32631]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:19:44.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:19:44.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:43 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/1233225932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:44.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:43 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/387973365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]: dispatch 2024-04-25T01:19:44.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:43 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]: dispatch 2024-04-25T01:19:44.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:43 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d73a4e5-3402-40e8-a625-53872c981f89"}]': finished 2024-04-25T01:19:44.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:43 smithi016 ceph-mon[29153]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:19:44.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:19:44.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:43 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/1233225932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:19:45.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:44 smithi016 ceph-mon[29153]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:45.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:44 smithi038 ceph-mon[32841]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:45.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:44 smithi156 ceph-mon[32631]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:47.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:46 smithi016 ceph-mon[29153]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:47.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:46 smithi038 ceph-mon[32841]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:47.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:46 smithi156 ceph-mon[32631]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:49.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:48 smithi016 ceph-mon[29153]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:49.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:48 smithi038 ceph-mon[32841]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:49.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:48 smithi156 ceph-mon[32631]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:51.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:50 smithi016 ceph-mon[29153]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:51.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:19:51.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:51.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:50 smithi016 ceph-mon[29153]: Deploying daemon osd.1 on smithi038 2024-04-25T01:19:51.195 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:50 smithi038 ceph-mon[32841]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:51.196 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:19:51.196 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:51.196 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:50 smithi038 ceph-mon[32841]: Deploying daemon osd.1 on smithi038 2024-04-25T01:19:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:50 smithi156 ceph-mon[32631]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:19:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:50 smithi156 ceph-mon[32631]: Deploying daemon osd.1 on smithi038 2024-04-25T01:19:53.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:52 smithi038 ceph-mon[32841]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:53.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:52 smithi156 ceph-mon[32631]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:53.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:52 smithi016 ceph-mon[29153]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:55.096 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:54 smithi038 ceph-mon[32841]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:55.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:54 smithi156 ceph-mon[32631]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:55.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:54 smithi016 ceph-mon[29153]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:56.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:56 smithi038 ceph-mon[32841]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:56.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:56.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:56.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:56 smithi156 ceph-mon[32631]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:56 smithi016 ceph-mon[29153]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:57.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:57.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:19:57.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.097 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:59 smithi038 ceph-mon[32841]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:59.097 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.098 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.098 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:59.098 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:59.098 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:19:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:59 smithi016 ceph-mon[29153]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:59.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:59.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:59.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:19:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.478+0000 7feba37fe700 1 -- 172.21.15.38:0/1629728980 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7febb40fe8f0 con 0x7feb9c0a2b90 2024-04-25T01:19:59.461 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 1 on host 'smithi038' 2024-04-25T01:19:59.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.480+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb9c0a2b90 msgr2=0x7feb9c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:59.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.480+0000 7febb8d17700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb9c0a2b90 0x7feb9c0a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feba40038f0 tx=0x7feba4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:19:59.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 msgr2=0x7febb4198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 0x7febb4198970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feba800c970 tx=0x7feba800cd30 comp rx=0 tx=0).stop 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 shutdown_connections 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb9c0a2b90 0x7feb9c0a5050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febb4102d70 0x7febb419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febb41011d0 0x7febb4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 --2- 172.21.15.38:0/1629728980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb41023d0 0x7febb4198970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 >> 172.21.15.38:0/1629728980 conn(0x7febb40fc750 msgr2=0x7febb40fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 shutdown_connections 2024-04-25T01:19:59.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:19:59.481+0000 7febb8d17700 1 -- 172.21.15.38:0/1629728980 wait complete. 2024-04-25T01:19:59.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:59 smithi156 ceph-mon[32631]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:19:59.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:19:59.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:19:59.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:19:59.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:19:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.055 DEBUG:teuthology.orchestra.run.smithi038:osd.1> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T01:20:00.058 INFO:tasks.cephadm:Deploying osd.2 on smithi156 with /dev/vg_nvme/lv_4... 2024-04-25T01:20:00.058 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:20:00.308 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:20:00.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:00 smithi038 ceph-mon[32841]: from='osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:20:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:00 smithi038 ceph-mon[32841]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:20:00.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:00 smithi038 ceph-mon[32841]: overall HEALTH_OK 2024-04-25T01:20:00.744 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.744 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:00.744 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.744 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:00 smithi156 ceph-mon[32631]: from='osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:20:00.744 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:00 smithi156 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:20:00.744 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:00 smithi156 ceph-mon[32631]: overall HEALTH_OK 2024-04-25T01:20:00.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:00.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:00.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:00 smithi016 ceph-mon[29153]: from='osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:20:00.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:00 smithi016 ceph-mon[29153]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:20:00.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:00 smithi016 ceph-mon[29153]: overall HEALTH_OK 2024-04-25T01:20:01.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-mon[32841]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:20:01.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-mon[32841]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:20:01.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-mon[32841]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:20:01.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:01.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-mon[32841]: from='osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:20:01.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-mon[32841]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:20:01.469 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1[37410]: 2024-04-25T01:20:01.464+0000 7f06da2b4700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:20:01.469 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1[37410]: 2024-04-25T01:20:01.468+0000 7f06d60a7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:20:01.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:01 smithi156 ceph-mon[32631]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:20:01.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:01 smithi156 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:20:01.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:01 smithi156 ceph-mon[32631]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:20:01.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:01.624 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:01 smithi156 ceph-mon[32631]: from='osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:20:01.624 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:01 smithi156 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:20:01.764 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:01 smithi016 ceph-mon[29153]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:20:01.765 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:01 smithi016 ceph-mon[29153]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:20:01.765 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:01 smithi016 ceph-mon[29153]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:20:01.765 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:01.765 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:01 smithi016 ceph-mon[29153]: from='osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:20:01.765 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:01 smithi016 ceph-mon[29153]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:20:02.189 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:20:02.189 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:20:02.189 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T01:20:02.189 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T01:20:02.189 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00829675 s, 1.3 GB/s 2024-04-25T01:20:02.189 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T01:20:02.538 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:02 smithi156 ceph-mon[32631]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:20:02.538 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:02 smithi156 ceph-mon[32631]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:20:02.538 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:02.538 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:02.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:02 smithi038 ceph-mon[32841]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:20:02.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:02 smithi038 ceph-mon[32841]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:20:02.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:02.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:02.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:02 smithi016 ceph-mon[29153]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:20:02.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:02 smithi016 ceph-mon[29153]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:20:02.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:02.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:02.888 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2024-04-25T01:20:03.127 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:20:03.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:20:03.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:20:03.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] boot 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:03.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] boot 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:03.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:03.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: osd.1 [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] boot 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:04.525 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:04 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi038 2024-04-25T01:20:04.525 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:04 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-25T01:20:04.525 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:04 smithi156 ceph-mon[32631]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:20:04.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 -- 172.21.15.156:0/1796585795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181034c0 msgr2=0x7f2118103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:04.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 --2- 172.21.15.156:0/1796585795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181034c0 0x7f2118103920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2108009990 tx=0x7f2108009ca0 comp rx=0 tx=0).stop 2024-04-25T01:20:04.718 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 -- 172.21.15.156:0/1796585795 shutdown_connections 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 --2- 172.21.15.156:0/1796585795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2118103e60 0x7f211810a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 --2- 172.21.15.156:0/1796585795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21181022c0 0x7f21181026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 --2- 172.21.15.156:0/1796585795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181034c0 0x7f2118103920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 -- 172.21.15.156:0/1796585795 >> 172.21.15.156:0/1796585795 conn(0x7f21180fd880 msgr2=0x7f21180ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 -- 172.21.15.156:0/1796585795 shutdown_connections 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.733+0000 7f211e326700 1 -- 172.21.15.156:0/1796585795 wait complete. 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 Processor -- start 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 -- start start 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 0x7f2118075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21181034c0 0x7f2118074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2118103e60 0x7f2118074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f211810dd90 con 0x7f21181034c0 2024-04-25T01:20:04.719 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f211810dc10 con 0x7f2118103e60 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211e326700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f211810df10 con 0x7f21181022c0 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f2117fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 0x7f2118075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f2117fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 0x7f2118075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:35992/0 (socket says 172.21.15.156:35992) 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f2117fff700 1 -- 172.21.15.156:0/2036593999 learned_addr learned my addr 172.21.15.156:0/2036593999 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f211c8c3700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2118103e60 0x7f2118074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.734+0000 7f21177fe700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21181034c0 0x7f2118074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f2117fff700 1 -- 172.21.15.156:0/2036593999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2118103e60 msgr2=0x7f2118074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f2117fff700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2118103e60 0x7f2118074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f2117fff700 1 -- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21181034c0 msgr2=0x7f2118074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f2117fff700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21181034c0 0x7f2118074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f2117fff700 1 -- 172.21.15.156:0/2036593999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2118075410 con 0x7f21181022c0 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f21177fe700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21181034c0 0x7f2118074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:20:04.720 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f2117fff700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 0x7f2118075fd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f210c00c990 tx=0x7f210c00cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:04.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c007b00 con 0x7f21181022c0 2024-04-25T01:20:04.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f210c007ca0 con 0x7f21181022c0 2024-04-25T01:20:04.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c018660 con 0x7f21181022c0 2024-04-25T01:20:04.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21181a2470 con 0x7f21181022c0 2024-04-25T01:20:04.721 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.735+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21181a2930 con 0x7f21181022c0 2024-04-25T01:20:04.721 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:04 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi038 2024-04-25T01:20:04.722 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:04 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-25T01:20:04.722 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:04 smithi038 ceph-mon[32841]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:20:04.722 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.736+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2118068f20 con 0x7f21181022c0 2024-04-25T01:20:04.722 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.737+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f210c00f450 con 0x7f21181022c0 2024-04-25T01:20:04.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.737+0000 7f21157fa700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21000a2c10 0x7f21000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:04.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.738+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f210c0cd6c0 con 0x7f21181022c0 2024-04-25T01:20:04.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.738+0000 7f21177fe700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21000a2c10 0x7f21000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:04.723 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.738+0000 7f21177fe700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21000a2c10 0x7f21000a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2108003920 tx=0x7f2108005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:04.725 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.742+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f210c014030 con 0x7f21181022c0 2024-04-25T01:20:04.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:04 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi038 2024-04-25T01:20:04.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:04 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-25T01:20:04.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:04 smithi016 ceph-mon[29153]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:20:04.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:04.944+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2118062670 con 0x7f21000a2c10 2024-04-25T01:20:05.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:05 smithi038 ceph-mon[32841]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:05.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:05.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:05.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:05.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:05 smithi156 ceph-mon[32631]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:05.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:05.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:05.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:05.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:05 smithi016 ceph-mon[29153]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:05.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:05.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:05.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:06.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:06 smithi038 ceph-mon[32841]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:06.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:06 smithi156 ceph-mon[32631]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:06.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:06 smithi016 ceph-mon[29153]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:07.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:07 smithi156 ceph-mon[32631]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:07.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:07 smithi016 ceph-mon[29153]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:07.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:07 smithi038 ceph-mon[32841]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:08.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:08 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/1591406375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]: dispatch 2024-04-25T01:20:08.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:08 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]: dispatch 2024-04-25T01:20:08.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:08 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]': finished 2024-04-25T01:20:08.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:08 smithi156 ceph-mon[32631]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:20:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:08.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:08 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/2165338535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:08.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:08 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/1591406375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]: dispatch 2024-04-25T01:20:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:08 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]: dispatch 2024-04-25T01:20:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:08 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]': finished 2024-04-25T01:20:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:08 smithi016 ceph-mon[29153]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:20:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:08.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:08 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/2165338535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:08.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:08 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/1591406375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]: dispatch 2024-04-25T01:20:08.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:08 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]: dispatch 2024-04-25T01:20:08.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:08 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffa52ec5-7d59-482c-bf75-87a3b7f21447"}]': finished 2024-04-25T01:20:08.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:08 smithi038 ceph-mon[32841]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:20:08.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:08.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:08 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/2165338535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:09.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:09 smithi156 ceph-mon[32631]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:09 smithi016 ceph-mon[29153]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:09.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:09 smithi038 ceph-mon[32841]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:10.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:10 smithi156 ceph-mon[32631]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:10.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:10 smithi016 ceph-mon[29153]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:10.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:10 smithi038 ceph-mon[32841]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:12.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:12 smithi016 ceph-mon[29153]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:12.898 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:12 smithi156 ceph-mon[32631]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:12.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:12 smithi038 ceph-mon[32841]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:14.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:14 smithi156 ceph-mon[32631]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:14 smithi016 ceph-mon[29153]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:14.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:14 smithi038 ceph-mon[32841]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:15.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:20:15.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:15.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:15 smithi156 ceph-mon[32631]: Deploying daemon osd.2 on smithi156 2024-04-25T01:20:15.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:20:15.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:15.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:15 smithi016 ceph-mon[29153]: Deploying daemon osd.2 on smithi156 2024-04-25T01:20:15.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:20:15.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:15.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:15 smithi038 ceph-mon[32841]: Deploying daemon osd.2 on smithi156 2024-04-25T01:20:16.710 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:16 smithi156 ceph-mon[32631]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:16 smithi016 ceph-mon[29153]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:16 smithi038 ceph-mon[32841]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:18.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:18 smithi156 ceph-mon[32631]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:18.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:18 smithi016 ceph-mon[29153]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:18.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:18 smithi038 ceph-mon[32841]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:21 smithi016 ceph-mon[29153]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:21.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:21 smithi038 ceph-mon[32841]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:21.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:21.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:21.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:21.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:21 smithi156 ceph-mon[32631]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:21.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:21.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:21.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:23.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.311+0000 7f21157fa700 1 -- 172.21.15.156:0/2036593999 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2118062670 con 0x7f21000a2c10 2024-04-25T01:20:23.295 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:23 smithi156 ceph-mon[32631]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:23.295 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 2 on host 'smithi156' 2024-04-25T01:20:23.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21000a2c10 msgr2=0x7f21000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:23.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21000a2c10 0x7f21000a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2108003920 tx=0x7f2108005fd0 comp rx=0 tx=0).stop 2024-04-25T01:20:23.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 msgr2=0x7f2118075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 0x7f2118075fd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f210c00c990 tx=0x7f210c00cca0 comp rx=0 tx=0).stop 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 shutdown_connections 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21000a2c10 0x7f21000a50d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2118103e60 0x7f2118074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21181022c0 0x7f2118075fd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 --2- 172.21.15.156:0/2036593999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21181034c0 0x7f2118074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 >> 172.21.15.156:0/2036593999 conn(0x7f21180fd880 msgr2=0x7f21181066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 shutdown_connections 2024-04-25T01:20:23.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:20:23.314+0000 7f211e326700 1 -- 172.21.15.156:0/2036593999 wait complete. 2024-04-25T01:20:23.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:23 smithi016 ceph-mon[29153]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:23.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:23 smithi038 ceph-mon[32841]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:23.915 DEBUG:teuthology.orchestra.run.smithi156:osd.2> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T01:20:23.917 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-04-25T01:20:23.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:20:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:20:24.187 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2[37788]: 2024-04-25T01:20:23.978+0000 7fd4f6c4b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T01:20:24.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:24 smithi038 ceph-mon[32841]: from='osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:20:24.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:24 smithi038 ceph-mon[32841]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:20:24.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:24 smithi156 ceph-mon[32631]: from='osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:20:24.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:24 smithi156 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:20:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:24 smithi016 ceph-mon[29153]: from='osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:20:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:24 smithi016 ceph-mon[29153]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:20:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:25 smithi156 ceph-mon[32631]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:25 smithi156 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:20:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:25 smithi156 ceph-mon[32631]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:20:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:25 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:25 smithi156 ceph-mon[32631]: from='osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:20:25.423 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:25 smithi156 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:20:25.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:25 smithi016 ceph-mon[29153]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:25.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:25 smithi016 ceph-mon[29153]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:20:25.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:25 smithi016 ceph-mon[29153]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:20:25.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:25 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:25.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:25 smithi016 ceph-mon[29153]: from='osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:20:25.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:25 smithi016 ceph-mon[29153]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:20:25.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:25 smithi038 ceph-mon[32841]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:25.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:25 smithi038 ceph-mon[32841]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:20:25.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:25 smithi038 ceph-mon[32841]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:20:25.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:25 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:25.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:25 smithi038 ceph-mon[32841]: from='osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:20:25.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:25 smithi038 ceph-mon[32841]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:20:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:20:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:20:26.257 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-25T01:20:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-25T01:20:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00877372 s, 1.2 GB/s 2024-04-25T01:20:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-25T01:20:26.503 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2[37788]: 2024-04-25T01:20:26.043+0000 7fd4ec2b0700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:20:26.503 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2[37788]: 2024-04-25T01:20:26.047+0000 7fd4e409b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:20:26.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:26 smithi016 ceph-mon[29153]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:26.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:26 smithi016 ceph-mon[29153]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:20:26.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:26 smithi016 ceph-mon[29153]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:20:26.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:26 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:26.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:26 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:26.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:26 smithi016 ceph-mon[29153]: from='osd.2 ' entity='osd.2' 2024-04-25T01:20:26.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:26 smithi038 ceph-mon[32841]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:26.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:26 smithi038 ceph-mon[32841]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:20:26.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:26 smithi038 ceph-mon[32841]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:20:26.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:26 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:26.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:26 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:26.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:26 smithi038 ceph-mon[32841]: from='osd.2 ' entity='osd.2' 2024-04-25T01:20:26.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-04-25T01:20:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-mon[32631]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-mon[32631]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:20:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-mon[32631]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:20:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:27.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:26 smithi156 ceph-mon[32631]: from='osd.2 ' entity='osd.2' 2024-04-25T01:20:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:20:27.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:20:27.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi156 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] boot 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:27.629 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:27.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:27.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:27.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi156 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] boot 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:20:27.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:27.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:27.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:27.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:27.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:20:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:20:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi156 2024-04-25T01:20:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: osd.2 [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] boot 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:28.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:28.699 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:28 smithi016 ceph-mon[29153]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:28.699 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:20:28.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 -- 172.21.15.16:0/3378901818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881023b0 msgr2=0x7f5388102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 --2- 172.21.15.16:0/3378901818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881023b0 0x7f5388102810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f53780099c0 tx=0x7f5378009cd0 comp rx=0 tx=0).stop 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 -- 172.21.15.16:0/3378901818 shutdown_connections 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 --2- 172.21.15.16:0/3378901818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5388102d50 0x7f5388109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 --2- 172.21.15.16:0/3378901818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53881011b0 0x7f53881015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 --2- 172.21.15.16:0/3378901818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881023b0 0x7f5388102810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 -- 172.21.15.16:0/3378901818 >> 172.21.15.16:0/3378901818 conn(0x7f53880fc750 msgr2=0x7f53880feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 -- 172.21.15.16:0/3378901818 shutdown_connections 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 -- 172.21.15.16:0/3378901818 wait complete. 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 Processor -- start 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.936+0000 7f538fef1700 1 -- start start 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 0x7f5388198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53881023b0 0x7f5388198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5388102d50 0x7f538819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:28.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f538810cc70 con 0x7f53881011b0 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f538810caf0 con 0x7f53881023b0 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f538810cdf0 con 0x7f5388102d50 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 0x7f5388198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 0x7f5388198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50818/0 (socket says 172.21.15.16:50818) 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 -- 172.21.15.16:0/3887870854 learned_addr learned my addr 172.21.15.16:0/3887870854 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538d48c700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53881023b0 0x7f5388198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538e48e700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5388102d50 0x7f538819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 -- 172.21.15.16:0/3887870854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5388102d50 msgr2=0x7f538819f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5388102d50 0x7f538819f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 -- 172.21.15.16:0/3887870854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53881023b0 msgr2=0x7f5388198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53881023b0 0x7f5388198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 -- 172.21.15.16:0/3887870854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53881a0050 con 0x7f53881011b0 2024-04-25T01:20:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538dc8d700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 0x7f5388198440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f538400d9b0 tx=0x7f538400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384017030 con 0x7f53881011b0 2024-04-25T01:20:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53881a0340 con 0x7f53881011b0 2024-04-25T01:20:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53840044d0 con 0x7f53881011b0 2024-04-25T01:20:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840039e0 con 0x7f53881011b0 2024-04-25T01:20:28.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.937+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53881a08d0 con 0x7f53881011b0 2024-04-25T01:20:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.938+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5388068f20 con 0x7f53881011b0 2024-04-25T01:20:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.940+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f538401a070 con 0x7f53881011b0 2024-04-25T01:20:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.940+0000 7f537effd700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f53740a2b90 0x7f53740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.940+0000 7f538d48c700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f53740a2b90 0x7f53740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.940+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f53840cd8e0 con 0x7f53881011b0 2024-04-25T01:20:28.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.941+0000 7f538d48c700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f53740a2b90 0x7f53740a5050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f53780038f0 tx=0x7f5378005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:28.944+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f538409bd00 con 0x7f53881011b0 2024-04-25T01:20:28.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:28 smithi038 ceph-mon[32841]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:28.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:28 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:20:29.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:28 smithi156 ceph-mon[32631]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:20:29.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:28 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:20:29.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:29.162+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f53880fe960 con 0x7f53740a2b90 2024-04-25T01:20:29.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:20:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:20:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:20:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:29.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:20:29.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:20:29.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:20:29.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:29.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:29.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:29.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:30.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:30.362 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:20:30.362 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:30.362 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48026]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:30.362 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48026]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:30.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:30 smithi016 ceph-mon[29153]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:30.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:30 smithi016 ceph-mon[29153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:20:30.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:20:30.846 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:30 smithi016 ceph-mon[29153]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:20:30.847 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48033]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:20:30.847 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48033]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:30.847 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48033]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:30.847 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:20:30 smithi016 sudo[48033]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:30 smithi038 ceph-mon[32841]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:30 smithi038 ceph-mon[32841]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:20:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:20:30.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:30 smithi038 ceph-mon[32841]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:20:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:30 smithi156 ceph-mon[32631]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:31.006 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:30 smithi156 ceph-mon[32631]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:20:31.006 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:20:31.006 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:30 smithi156 ceph-mon[32631]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:20:31.468 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41260]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:20:31.468 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41260]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:31.468 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41260]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:31.468 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41260]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:31.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:31 smithi016 ceph-mon[29153]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:20:31.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:31.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:31.968 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41263]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:20:31.969 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:31.969 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41263]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:31.969 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:20:31 smithi038 sudo[41263]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:31.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:31 smithi038 ceph-mon[32841]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:20:31.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:31.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:32.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:31 smithi156 ceph-mon[32631]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:20:32.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:32.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:32.003 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:31 smithi156 sudo[41401]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:20:32.003 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:31 smithi156 sudo[41401]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:32.003 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:31 smithi156 sudo[41401]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:32.003 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:31 smithi156 sudo[41401]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:32.503 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:32 smithi156 sudo[41404]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:20:32.503 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:32 smithi156 sudo[41404]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:32.503 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:32 smithi156 sudo[41404]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:32.503 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:20:32 smithi156 sudo[41404]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:32.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48135]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:20:32.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48135]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:32.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48135]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:32.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48135]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:33.004+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5384097c70 con 0x7f53881011b0 2024-04-25T01:20:33.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48138]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:20:33.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48138]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:33.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48138]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:33.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:32 smithi016 sudo[48138]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:33.474 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41407]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:20:33.474 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41407]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:33.474 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41407]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:33.474 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41407]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41410]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41410]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41410]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 sudo[41410]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: Cluster is now healthy 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3715231609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c945cb2-a2ff-473b-98a7-2e462b7c59b6"}]: dispatch 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3715231609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c945cb2-a2ff-473b-98a7-2e462b7c59b6"}]': finished 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:20:33.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:33.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:33.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:33 smithi156 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:33.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: Cluster is now healthy 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3715231609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c945cb2-a2ff-473b-98a7-2e462b7c59b6"}]: dispatch 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3715231609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c945cb2-a2ff-473b-98a7-2e462b7c59b6"}]': finished 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:33 smithi016 ceph-mon[29153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:33.927 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:20:33.927 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: Cluster is now healthy 2024-04-25T01:20:33.928 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3715231609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c945cb2-a2ff-473b-98a7-2e462b7c59b6"}]: dispatch 2024-04-25T01:20:33.928 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3715231609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c945cb2-a2ff-473b-98a7-2e462b7c59b6"}]': finished 2024-04-25T01:20:33.928 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:20:33.928 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:33.928 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:33.928 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 ceph-mon[32841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 sudo[41266]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:20:34.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 sudo[41266]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:34.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 sudo[41266]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:34.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:33 smithi038 sudo[41266]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:34.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 sudo[41269]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 sudo[41269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 sudo[41269]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 sudo[41269]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:20:34.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2798387557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:34.720 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:34 smithi038 ceph-mon[32841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2798387557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:34.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:34 smithi156 ceph-mon[32631]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2798387557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:34.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:20:34.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:20:34.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:20:34.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:20:34.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:34.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:34 smithi016 ceph-mon[29153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:20:36.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:36 smithi016 ceph-mon[29153]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:36.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:36 smithi038 ceph-mon[32841]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:37.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:36 smithi156 ceph-mon[32631]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:38.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:38 smithi016 ceph-mon[29153]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:38.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:38 smithi038 ceph-mon[32841]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:39.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:38 smithi156 ceph-mon[32631]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:40 smithi016 ceph-mon[29153]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-25T01:20:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:20:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:40 smithi016 ceph-mon[29153]: Deploying daemon osd.3 on smithi016 2024-04-25T01:20:40.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:40 smithi038 ceph-mon[32841]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-25T01:20:40.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:20:40.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:40.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:40 smithi038 ceph-mon[32841]: Deploying daemon osd.3 on smithi016 2024-04-25T01:20:41.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:40 smithi156 ceph-mon[32631]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-25T01:20:41.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:20:41.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:41.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:40 smithi156 ceph-mon[32631]: Deploying daemon osd.3 on smithi016 2024-04-25T01:20:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:42 smithi016 ceph-mon[29153]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-25T01:20:42.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:42 smithi038 ceph-mon[32841]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-25T01:20:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:42 smithi156 ceph-mon[32631]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-25T01:20:44.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:44 smithi016 ceph-mon[29153]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:20:44.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:44 smithi038 ceph-mon[32841]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:20:45.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:44 smithi156 ceph-mon[32631]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:20:47.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:46 smithi016 ceph-mon[29153]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:47.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:46 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:47.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:46 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:47.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:46 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:47.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:46 smithi038 ceph-mon[32841]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:47.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:46 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:47.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:46 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:47.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:46 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:47.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:46 smithi156 ceph-mon[32631]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:47.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:47.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:47.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:48 smithi016 ceph-mon[29153]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:49.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:48 smithi038 ceph-mon[32841]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:49.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:49.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:48 smithi156 ceph-mon[32631]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:49.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:49.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.723+0000 7f537effd700 1 -- 172.21.15.16:0/3887870854 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f53880fe960 con 0x7f53740a2b90 2024-04-25T01:20:49.706 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f53740a2b90 msgr2=0x7f53740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f53740a2b90 0x7f53740a5050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f53780038f0 tx=0x7f5378005fb0 comp rx=0 tx=0).stop 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 msgr2=0x7f5388198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 0x7f5388198440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f538400d9b0 tx=0x7f538400dd70 comp rx=0 tx=0).stop 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 shutdown_connections 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f53740a2b90 0x7f53740a5050 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5388102d50 0x7f538819f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53881011b0 0x7f5388198440 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 --2- 172.21.15.16:0/3887870854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53881023b0 0x7f5388198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:49.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 >> 172.21.15.16:0/3887870854 conn(0x7f53880fc750 msgr2=0x7f53880fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:49.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 shutdown_connections 2024-04-25T01:20:49.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:20:49.726+0000 7f538fef1700 1 -- 172.21.15.16:0/3887870854 wait complete. 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:20:50.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:20:50.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:50.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:50.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:50.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:50.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:20:50.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:20:50.503 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T01:20:50.505 INFO:tasks.cephadm:Deploying osd.4 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-25T01:20:50.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:20:50.763 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:20:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:50 smithi016 ceph-mon[29153]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:50 smithi016 ceph-mon[29153]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:20:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:50 smithi016 ceph-mon[29153]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:20:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:50 smithi016 ceph-mon[29153]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:20:51.000 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:51.116 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:50 smithi038 ceph-mon[32841]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:51.116 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:50 smithi038 ceph-mon[32841]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:20:51.116 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:50 smithi038 ceph-mon[32841]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:20:51.116 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:50 smithi038 ceph-mon[32841]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:20:51.117 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:50 smithi156 ceph-mon[32631]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-25T01:20:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:50 smithi156 ceph-mon[32631]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:20:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:50 smithi156 ceph-mon[32631]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:20:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:50 smithi156 ceph-mon[32631]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:20:51.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:51.771 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:20:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:20:51.692+0000 7f865fb5a700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:20:51.771 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:20:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:20:51.698+0000 7f865b94d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:20:52.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:52 smithi038 ceph-mon[32841]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:52.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:52 smithi038 ceph-mon[32841]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:20:52.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:52 smithi038 ceph-mon[32841]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:20:52.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:52.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:52.815 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:20:52.815 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:20:52.815 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-25T01:20:52.815 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-25T01:20:52.816 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00822187 s, 1.3 GB/s 2024-04-25T01:20:52.816 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-25T01:20:53.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:52 smithi156 ceph-mon[32631]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:53.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:52 smithi156 ceph-mon[32631]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:20:53.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:52 smithi156 ceph-mon[32631]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:20:53.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:53.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:53.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:52 smithi016 ceph-mon[29153]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:20:53.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:52 smithi016 ceph-mon[29153]: from='osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:20:53.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:52 smithi016 ceph-mon[29153]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:20:53.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:53.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:53.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-25T01:20:53.724 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:20:53.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:20:53.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] boot 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:53.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:53 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] boot 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:53.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:53.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:53.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:53.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:53.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:53 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: osd.3 [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] boot 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:20:54.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:54.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:54.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:20:54.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:53 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:20:54.758 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:54 smithi038 ceph-mon[32841]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:54.759 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:54 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi016 2024-04-25T01:20:54.759 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:54 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi016 to 8546M 2024-04-25T01:20:54.759 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:54 smithi038 ceph-mon[32841]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:20:55.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:54 smithi156 ceph-mon[32631]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:55.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:54 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi016 2024-04-25T01:20:55.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:54 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi016 to 8546M 2024-04-25T01:20:55.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:54 smithi156 ceph-mon[32631]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:20:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:54 smithi016 ceph-mon[29153]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:54 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi016 2024-04-25T01:20:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:54 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi016 to 8546M 2024-04-25T01:20:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:54 smithi016 ceph-mon[29153]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 -- 172.21.15.38:0/2172046217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8107f80 msgr2=0x7fb5f810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/2172046217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8107f80 0x7fb5f810a3f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000b0a0 tx=0x7fb5f000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 -- 172.21.15.38:0/2172046217 shutdown_connections 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/2172046217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8107f80 0x7fb5f810a3f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/2172046217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8102d20 0x7fb5f8105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/2172046217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f8105650 0x7fb5f8107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 -- 172.21.15.38:0/2172046217 >> 172.21.15.38:0/2172046217 conn(0x7fb5f80fc730 msgr2=0x7fb5f80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 -- 172.21.15.38:0/2172046217 shutdown_connections 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 -- 172.21.15.38:0/2172046217 wait complete. 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.320+0000 7fb5fdb34700 1 Processor -- start 2024-04-25T01:20:55.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 -- start start 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 0x7fb5f8102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8105650 0x7fb5f8100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f8107f80 0x7fb5f8101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f810dd70 con 0x7fb5f8107f80 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f810dbf0 con 0x7fb5f8102d20 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5fdb34700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f810def0 con 0x7fb5f8105650 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 0x7fb5f8102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 0x7fb5f8102680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48928/0 (socket says 172.21.15.38:48928) 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 -- 172.21.15.38:0/182871398 learned_addr learned my addr 172.21.15.38:0/182871398 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f7fff700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f8107f80 0x7fb5f8101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f6ffd700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8105650 0x7fb5f8100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 -- 172.21.15.38:0/182871398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8105650 msgr2=0x7fb5f8100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:55.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8105650 0x7fb5f8100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 -- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f8107f80 msgr2=0x7fb5f8101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f8107f80 0x7fb5f8101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 -- 172.21.15.38:0/182871398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f8101ac0 con 0x7fb5f8102d20 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f6ffd700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8105650 0x7fb5f8100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.321+0000 7fb5f77fe700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 0x7fb5f8102680 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c920 tx=0x7fb5e800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.322+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8007b00 con 0x7fb5f8102d20 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.322+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5e8004ce0 con 0x7fb5f8102d20 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.322+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8005550 con 0x7fb5f8102d20 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.322+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f81a2550 con 0x7fb5f8102d20 2024-04-25T01:20:55.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.322+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f81a2ae0 con 0x7fb5f8102d20 2024-04-25T01:20:55.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.323+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f8068f20 con 0x7fb5f8102d20 2024-04-25T01:20:55.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.324+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb5e8004750 con 0x7fb5f8102d20 2024-04-25T01:20:55.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.325+0000 7fb5f4ff9700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb5e00a2c20 0x7fb5e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:20:55.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.325+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7fb5e80cdd30 con 0x7fb5f8102d20 2024-04-25T01:20:55.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.325+0000 7fb5f6ffd700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb5e00a2c20 0x7fb5e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:20:55.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.325+0000 7fb5f6ffd700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb5e00a2c20 0x7fb5e00a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec007900 tx=0x7fb5ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:20:55.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.330+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb5e8014030 con 0x7fb5f8102d20 2024-04-25T01:20:55.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:20:55.545+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb5f8062670 con 0x7fb5e00a2c20 2024-04-25T01:20:55.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:55 smithi038 ceph-mon[32841]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:20:55.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:55.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:55.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:56.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:55 smithi156 ceph-mon[32631]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:20:56.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:55 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:56.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:55 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:56.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:55 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:55 smithi016 ceph-mon[29153]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:20:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:55 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:20:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:55 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:20:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:55 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:20:56.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:56 smithi038 ceph-mon[32841]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:56.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:56 smithi038 ceph-mon[32841]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:56.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:56 smithi038 ceph-mon[32841]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:20:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:56 smithi156 ceph-mon[32631]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:56 smithi156 ceph-mon[32631]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:57.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:56 smithi156 ceph-mon[32631]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:20:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:56 smithi016 ceph-mon[29153]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:20:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:56 smithi016 ceph-mon[29153]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:56 smithi016 ceph-mon[29153]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:20:58.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:57 smithi156 ceph-mon[32631]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:20:58.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:57 smithi016 ceph-mon[29153]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:20:58.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:57 smithi038 ceph-mon[32841]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:20:59.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:58 smithi156 ceph-mon[32631]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:59.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:58 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/1925277697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]: dispatch 2024-04-25T01:20:59.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:58 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]: dispatch 2024-04-25T01:20:59.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:58 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]': finished 2024-04-25T01:20:59.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:58 smithi156 ceph-mon[32631]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:20:59.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:58 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:20:59.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:58 smithi016 ceph-mon[29153]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:59.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:58 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/1925277697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]: dispatch 2024-04-25T01:20:59.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:58 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]: dispatch 2024-04-25T01:20:59.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:58 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]': finished 2024-04-25T01:20:59.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:58 smithi016 ceph-mon[29153]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:20:59.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:58 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:20:59.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:58 smithi038 ceph-mon[32841]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:20:59.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:58 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/1925277697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]: dispatch 2024-04-25T01:20:59.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:58 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]: dispatch 2024-04-25T01:20:59.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:58 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e91cb88-1603-4183-b475-1ab7df79597a"}]': finished 2024-04-25T01:20:59.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:58 smithi038 ceph-mon[32841]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:20:59.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:58 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:00.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:20:59 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/3743423671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:20:59 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/3743423671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:00.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:20:59 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/3743423671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:01.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:00 smithi156 ceph-mon[32631]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-25T01:21:01.013 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:00 smithi016 ceph-mon[29153]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-25T01:21:01.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:00 smithi038 ceph-mon[32841]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-25T01:21:03.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:02 smithi156 ceph-mon[32631]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:21:03.102 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:02 smithi038 ceph-mon[32841]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:21:03.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:02 smithi016 ceph-mon[29153]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:21:05.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:04 smithi156 ceph-mon[32631]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T01:21:05.009 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:04 smithi038 ceph-mon[32841]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T01:21:05.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:04 smithi016 ceph-mon[29153]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T01:21:07.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:06 smithi156 ceph-mon[32631]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-25T01:21:07.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:21:07.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:07.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:06 smithi156 ceph-mon[32631]: Deploying daemon osd.4 on smithi038 2024-04-25T01:21:07.051 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:06 smithi038 ceph-mon[32841]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-25T01:21:07.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:21:07.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:07.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:06 smithi038 ceph-mon[32841]: Deploying daemon osd.4 on smithi038 2024-04-25T01:21:07.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:06 smithi016 ceph-mon[29153]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-25T01:21:07.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:21:07.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:07.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:06 smithi016 ceph-mon[29153]: Deploying daemon osd.4 on smithi038 2024-04-25T01:21:08.945 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:08 smithi038 ceph-mon[32841]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:09.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:08 smithi156 ceph-mon[32631]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:09.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:08 smithi016 ceph-mon[29153]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:10.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:10 smithi038 ceph-mon[32841]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T01:21:11.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:10 smithi156 ceph-mon[32631]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T01:21:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:10 smithi016 ceph-mon[29153]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-25T01:21:13.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:12 smithi038 ceph-mon[32841]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:13.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:13.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:13.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:13.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:12 smithi156 ceph-mon[32631]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:13.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:13.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:13.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:13.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:12 smithi016 ceph-mon[29153]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:13.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:13.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:13.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:15.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:14 smithi038 ceph-mon[32841]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:15.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:14 smithi156 ceph-mon[32631]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:15.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:14 smithi016 ceph-mon[29153]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:15.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.616+0000 7fb5f4ff9700 1 -- 172.21.15.38:0/182871398 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb5f8062670 con 0x7fb5e00a2c20 2024-04-25T01:21:15.599 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 4 on host 'smithi038' 2024-04-25T01:21:15.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.619+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb5e00a2c20 msgr2=0x7fb5e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:15.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.619+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb5e00a2c20 0x7fb5e00a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec007900 tx=0x7fb5ec008040 comp rx=0 tx=0).stop 2024-04-25T01:21:15.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.619+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 msgr2=0x7fb5f8102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.619+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 0x7fb5f8102680 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c920 tx=0x7fb5e800cc30 comp rx=0 tx=0).stop 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 shutdown_connections 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb5e00a2c20 0x7fb5e00a50e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5f8107f80 0x7fb5f8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5f8102d20 0x7fb5f8102680 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 --2- 172.21.15.38:0/182871398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5f8105650 0x7fb5f8100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 >> 172.21.15.38:0/182871398 conn(0x7fb5f80fc730 msgr2=0x7fb5f8103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 shutdown_connections 2024-04-25T01:21:15.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:21:15.620+0000 7fb5fdb34700 1 -- 172.21.15.38:0/182871398 wait complete. 2024-04-25T01:21:16.263 DEBUG:teuthology.orchestra.run.smithi038:osd.4> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T01:21:16.265 INFO:tasks.cephadm:Deploying osd.5 on smithi156 with /dev/vg_nvme/lv_3... 2024-04-25T01:21:16.266 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:21:16.511 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: from='osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: from='osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:21:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:21:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: from='osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:17 smithi016 ceph-mon[29153]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:21:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:17 smithi016 ceph-mon[29153]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:21:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:17 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:17 smithi016 ceph-mon[29153]: from='osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:21:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:17 smithi016 ceph-mon[29153]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:21:17.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:17 smithi156 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:21:17.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:17 smithi156 ceph-mon[32631]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:21:17.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:17 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:17.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:17 smithi156 ceph-mon[32631]: from='osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:21:17.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:17 smithi156 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:21:17.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-mon[32841]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:21:17.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-mon[32841]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:21:17.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:17.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-mon[32841]: from='osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:21:17.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-mon[32841]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:21:17.968 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:21:17.610+0000 7f26aac4e700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:21:17.969 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:21:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:21:17.616+0000 7f26a6a41700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:18.416 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:21:18.417 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:21:18.417 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T01:21:18.417 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T01:21:18.417 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00863793 s, 1.2 GB/s 2024-04-25T01:21:18.417 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T01:21:18.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:18 smithi156 ceph-mon[32631]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:18.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:18 smithi156 ceph-mon[32631]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:21:18.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:18 smithi156 ceph-mon[32631]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:21:18.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:18 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:18.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:18 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:18.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:18 smithi016 ceph-mon[29153]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:18 smithi016 ceph-mon[29153]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:21:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:18 smithi016 ceph-mon[29153]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:21:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:18 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:18.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:18 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:18.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:18 smithi038 ceph-mon[32841]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:18.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:18 smithi038 ceph-mon[32841]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:21:18.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:18 smithi038 ceph-mon[32841]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:21:18.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:18 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:18.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:18 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:19.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2024-04-25T01:21:19.362 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:21:19.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:19 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:21:19.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:19 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:21:19.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:19 smithi156 ceph-mon[32631]: osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] boot 2024-04-25T01:21:19.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:19 smithi156 ceph-mon[32631]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:21:19.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:19.849 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:19 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:21:19.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:19 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:21:19.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:19 smithi038 ceph-mon[32841]: osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] boot 2024-04-25T01:21:19.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:19 smithi038 ceph-mon[32841]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:21:19.850 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:19 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:21:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:19 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:21:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:19 smithi016 ceph-mon[29153]: osd.4 [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] boot 2024-04-25T01:21:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:19 smithi016 ceph-mon[29153]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:21:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:21:20.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi038 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:20.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:20.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:20.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:20.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi038 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:20.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:20.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:21:20.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:21:20.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi038 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:20.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:20.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 -- 172.21.15.156:0/525384547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105720 msgr2=0x7ff7c4107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:20.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 --2- 172.21.15.156:0/525384547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105720 0x7ff7c4107b10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009990 tx=0x7ff7c0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:21:20.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 -- 172.21.15.156:0/525384547 shutdown_connections 2024-04-25T01:21:20.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 --2- 172.21.15.156:0/525384547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c4108050 0x7ff7c410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 --2- 172.21.15.156:0/525384547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c4102df0 0x7ff7c41051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 --2- 172.21.15.156:0/525384547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105720 0x7ff7c4107b10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 -- 172.21.15.156:0/525384547 >> 172.21.15.156:0/525384547 conn(0x7ff7c40fc760 msgr2=0x7ff7c40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 -- 172.21.15.156:0/525384547 shutdown_connections 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.011+0000 7ff7cb607700 1 -- 172.21.15.156:0/525384547 wait complete. 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 Processor -- start 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 -- start start 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c4102df0 0x7ff7c419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c4105720 0x7ff7c419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 0x7ff7c41a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c410ddf0 con 0x7ff7c4102df0 2024-04-25T01:21:20.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c410dc70 con 0x7ff7c4105720 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7cb607700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c410df70 con 0x7ff7c4108050 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 0x7ff7c41a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 0x7ff7c41a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59924/0 (socket says 172.21.15.156:59924) 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 -- 172.21.15.156:0/191893545 learned_addr learned my addr 172.21.15.156:0/191893545 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c93a3700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c4102df0 0x7ff7c419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c8ba2700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c4105720 0x7ff7c419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 -- 172.21.15.156:0/191893545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c4105720 msgr2=0x7ff7c419ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c4105720 0x7ff7c419ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 -- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c4102df0 msgr2=0x7ff7c419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c4102df0 0x7ff7c419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.012+0000 7ff7c9ba4700 1 -- 172.21.15.156:0/191893545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c41a2380 con 0x7ff7c4108050 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.013+0000 7ff7c9ba4700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 0x7ff7c41a1ce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00b9a0 tx=0x7ff7bc00bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:20.997 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.013+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc00d850 con 0x7ff7c4108050 2024-04-25T01:21:20.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.013+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7bc00e420 con 0x7ff7c4108050 2024-04-25T01:21:20.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.013+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc017640 con 0x7ff7c4108050 2024-04-25T01:21:20.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.013+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c41a2670 con 0x7ff7c4108050 2024-04-25T01:21:20.998 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.013+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c41a2c30 con 0x7ff7c4108050 2024-04-25T01:21:20.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.014+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c4068f20 con 0x7ff7c4108050 2024-04-25T01:21:20.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.015+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff7bc00e5d0 con 0x7ff7c4108050 2024-04-25T01:21:20.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.015+0000 7ff7ba7fc700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff7b00a2ca0 0x7ff7b00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:20.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.015+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7ff7bc0cdc10 con 0x7ff7c4108050 2024-04-25T01:21:20.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.015+0000 7ff7c93a3700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff7b00a2ca0 0x7ff7b00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:20.999 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.016+0000 7ff7c93a3700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff7b00a2ca0 0x7ff7b00a5160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009a10 tx=0x7ff7b4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:21.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.019+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7bc013030 con 0x7ff7c4108050 2024-04-25T01:21:21.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:21.224+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff7c4062670 con 0x7ff7b00a2ca0 2024-04-25T01:21:21.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:21 smithi016 ceph-mon[29153]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:21.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:21.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:21.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:21.961 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:21 smithi156 ceph-mon[32631]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:21.961 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:21.961 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:21.961 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:21.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:21 smithi038 ceph-mon[32841]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:21.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:21.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:21.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:22.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:22 smithi016 ceph-mon[29153]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:22.967 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:22 smithi156 ceph-mon[32631]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:22 smithi038 ceph-mon[32841]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:24.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:24 smithi038 ceph-mon[32841]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:24.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:24 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/3525922500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]: dispatch 2024-04-25T01:21:24.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:24 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]: dispatch 2024-04-25T01:21:24.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:24 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]': finished 2024-04-25T01:21:24.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:24 smithi038 ceph-mon[32841]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:21:24.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:24 smithi156 ceph-mon[32631]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:24 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/3525922500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]: dispatch 2024-04-25T01:21:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:24 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]: dispatch 2024-04-25T01:21:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:24 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]': finished 2024-04-25T01:21:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:24 smithi156 ceph-mon[32631]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:21:25.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:24 smithi016 ceph-mon[29153]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:24 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/3525922500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]: dispatch 2024-04-25T01:21:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:24 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]: dispatch 2024-04-25T01:21:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:24 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "451201e8-0fc6-4319-b7d9-9be4425d6384"}]': finished 2024-04-25T01:21:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:24 smithi016 ceph-mon[29153]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:21:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:25.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:25 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/4226923846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:26.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:25 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/4226923846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:26.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:25 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/4226923846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:26.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:26 smithi038 ceph-mon[32841]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:27.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:26 smithi156 ceph-mon[32631]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:26 smithi016 ceph-mon[29153]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:28.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:28 smithi038 ceph-mon[32841]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:28.978 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:28 smithi156 ceph-mon[32631]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:29.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:28 smithi016 ceph-mon[29153]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:30 smithi038 ceph-mon[32841]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:31.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:30 smithi156 ceph-mon[32631]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:31.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:30 smithi016 ceph-mon[29153]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:31.957 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:21:31.957 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:31.957 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:31 smithi156 ceph-mon[32631]: Deploying daemon osd.5 on smithi156 2024-04-25T01:21:31.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:21:31.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:31.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:31 smithi038 ceph-mon[32841]: Deploying daemon osd.5 on smithi156 2024-04-25T01:21:32.013 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:21:32.013 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:32.013 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:31 smithi016 ceph-mon[29153]: Deploying daemon osd.5 on smithi156 2024-04-25T01:21:32.937 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:32 smithi156 ceph-mon[32631]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:32.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:32 smithi038 ceph-mon[32841]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:32 smithi016 ceph-mon[29153]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:34.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:34 smithi156 ceph-mon[32631]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:34.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:34 smithi038 ceph-mon[32841]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:34 smithi016 ceph-mon[29153]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:36.922 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:36 smithi156 ceph-mon[32631]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:36.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:36 smithi038 ceph-mon[32841]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:36 smithi016 ceph-mon[29153]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:38.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:38.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:38.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:38.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:38.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:38.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:38.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:38.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:38.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:39.238 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:39 smithi156 ceph-mon[32631]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:39.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:39 smithi016 ceph-mon[29153]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:39.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:39 smithi038 ceph-mon[32841]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:40.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.444+0000 7ff7ba7fc700 1 -- 172.21.15.156:0/191893545 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff7c4062670 con 0x7ff7b00a2ca0 2024-04-25T01:21:40.427 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 5 on host 'smithi156' 2024-04-25T01:21:40.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff7b00a2ca0 msgr2=0x7ff7b00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff7b00a2ca0 0x7ff7b00a5160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009a10 tx=0x7ff7b4009190 comp rx=0 tx=0).stop 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 msgr2=0x7ff7c41a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 0x7ff7c41a1ce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00b9a0 tx=0x7ff7bc00bcb0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 shutdown_connections 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff7b00a2ca0 0x7ff7b00a5160 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4108050 0x7ff7c41a1ce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7c4102df0 0x7ff7c419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 --2- 172.21.15.156:0/191893545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7c4105720 0x7ff7c419ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 >> 172.21.15.156:0/191893545 conn(0x7ff7c40fc760 msgr2=0x7ff7c4102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 shutdown_connections 2024-04-25T01:21:40.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:21:40.447+0000 7ff7cb607700 1 -- 172.21.15.156:0/191893545 wait complete. 2024-04-25T01:21:41.119 DEBUG:teuthology.orchestra.run.smithi156:osd.5> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T01:21:41.122 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-04-25T01:21:41.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:21:41.384 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: from='osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: from='osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: from='osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:41.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:42.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:42 smithi016 ceph-mon[29153]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:21:42.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:42 smithi016 ceph-mon[29153]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:21:42.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:42.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:42 smithi016 ceph-mon[29153]: from='osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:21:42.643 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:42 smithi016 ceph-mon[29153]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:21:42.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:42 smithi038 ceph-mon[32841]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:21:42.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:42 smithi038 ceph-mon[32841]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:21:42.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:42.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:42 smithi038 ceph-mon[32841]: from='osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:21:42.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:42 smithi038 ceph-mon[32841]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:21:42.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:21:42.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-mon[32631]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:21:42.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:42.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-mon[32631]: from='osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:21:42.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:21:42.753 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:21:42.415+0000 7f9640f29700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:21:42.753 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:21:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:21:42.421+0000 7f963cd1c700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:21:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:43 smithi016 ceph-mon[29153]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:43 smithi016 ceph-mon[29153]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:21:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:43 smithi016 ceph-mon[29153]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:21:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:43.610 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:21:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:21:43.611 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-25T01:21:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-25T01:21:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0086052 s, 1.2 GB/s 2024-04-25T01:21:43.611 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-25T01:21:43.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:43 smithi038 ceph-mon[32841]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:43.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:43 smithi038 ceph-mon[32841]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:21:43.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:43 smithi038 ceph-mon[32841]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:21:43.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:43.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:43.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:43 smithi156 ceph-mon[32631]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:21:43.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:43 smithi156 ceph-mon[32631]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:21:43.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:43 smithi156 ceph-mon[32631]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:21:43.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:43.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:44.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-04-25T01:21:44.673 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:21:44.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:44 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:21:44.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:44 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:21:44.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:44 smithi038 ceph-mon[32841]: osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] boot 2024-04-25T01:21:44.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:44 smithi038 ceph-mon[32841]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:21:44.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:44 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:44.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:44 smithi038 ceph-mon[32841]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:21:44.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:44 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:21:44.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:44 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:21:44.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:44 smithi156 ceph-mon[32631]: osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] boot 2024-04-25T01:21:44.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:44 smithi156 ceph-mon[32631]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:21:44.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:44 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:44.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:44 smithi156 ceph-mon[32631]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:21:44.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:44 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:21:44.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:44 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:21:44.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:44 smithi016 ceph-mon[29153]: osd.5 [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] boot 2024-04-25T01:21:44.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:44 smithi016 ceph-mon[29153]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:21:44.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:44 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:21:44.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:44 smithi016 ceph-mon[29153]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:45.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:45.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:45.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:45.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:45 smithi038 ceph-mon[32841]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:21:45.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:45 smithi156 ceph-mon[32631]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:21:45.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:21:45.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:21:45.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:45 smithi016 ceph-mon[29153]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:21:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.532+0000 7f5112cc3700 1 -- 172.21.15.16:0/3106250432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1023b0 msgr2=0x7f510c102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.532+0000 7f5112cc3700 1 --2- 172.21.15.16:0/3106250432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1023b0 0x7f510c102810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f50f80099c0 tx=0x7f50f8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:46 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi156 2024-04-25T01:21:46.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:46 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-25T01:21:46.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:46 smithi016 ceph-mon[29153]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:21:46.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 -- 172.21.15.16:0/3106250432 shutdown_connections 2024-04-25T01:21:46.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 --2- 172.21.15.16:0/3106250432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c102d50 0x7f510c109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 --2- 172.21.15.16:0/3106250432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f510c1011b0 0x7f510c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 --2- 172.21.15.16:0/3106250432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1023b0 0x7f510c102810 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 -- 172.21.15.16:0/3106250432 >> 172.21.15.16:0/3106250432 conn(0x7f510c0fc750 msgr2=0x7f510c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 -- 172.21.15.16:0/3106250432 shutdown_connections 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 -- 172.21.15.16:0/3106250432 wait complete. 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 Processor -- start 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.533+0000 7f5112cc3700 1 -- start start 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5112cc3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 0x7f510c06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5112cc3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c1023b0 0x7f510c19d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5112cc3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f510c102d50 0x7f510c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5112cc3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f510c10cc30 con 0x7f510c1011b0 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5112cc3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f510c10cab0 con 0x7f510c102d50 2024-04-25T01:21:46.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5112cc3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f510c10cdb0 con 0x7f510c1023b0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 0x7f510c06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 0x7f510c06b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43580/0 (socket says 172.21.15.16:43580) 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 -- 172.21.15.16:0/2578234291 learned_addr learned my addr 172.21.15.16:0/2578234291 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5111260700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f510c102d50 0x7f510c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f510bfff700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c1023b0 0x7f510c19d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 -- 172.21.15.16:0/2578234291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c1023b0 msgr2=0x7f510c19d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c1023b0 0x7f510c19d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 -- 172.21.15.16:0/2578234291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f510c102d50 msgr2=0x7f510c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f510c102d50 0x7f510c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 -- 172.21.15.16:0/2578234291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c19ffb0 con 0x7f510c1011b0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5110a5f700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 0x7f510c06b4c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f510000d980 tx=0x7f510000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.534+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100017030 con 0x7f510c1011b0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.535+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51000044d0 con 0x7f510c1011b0 2024-04-25T01:21:46.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.535+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510c1a02a0 con 0x7f510c1011b0 2024-04-25T01:21:46.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.535+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51000039e0 con 0x7f510c1011b0 2024-04-25T01:21:46.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.535+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f510c1a0830 con 0x7f510c1011b0 2024-04-25T01:21:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.536+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f510c068f20 con 0x7f510c1011b0 2024-04-25T01:21:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.537+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f510001a070 con 0x7f510c1011b0 2024-04-25T01:21:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.537+0000 7f5109ffb700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f50fc0a2ca0 0x7f50fc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:21:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.538+0000 7f510bfff700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f50fc0a2ca0 0x7f50fc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:21:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.538+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f51000cd260 con 0x7f510c1011b0 2024-04-25T01:21:46.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.538+0000 7f510bfff700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f50fc0a2ca0 0x7f50fc0a5160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f510c06ba40 tx=0x7f50f8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:21:46.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.542+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f510009b080 con 0x7f510c1011b0 2024-04-25T01:21:46.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:46 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi156 2024-04-25T01:21:46.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:46 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-25T01:21:46.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:46 smithi038 ceph-mon[32841]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:21:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:21:46.758+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f510c062670 con 0x7f50fc0a2ca0 2024-04-25T01:21:46.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:46 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi156 2024-04-25T01:21:46.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:46 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-25T01:21:46.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:46 smithi156 ceph-mon[32631]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:21:47.641 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:47 smithi016 ceph-mon[29153]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:47.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:47 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:47.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:47 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:47.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:47 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:47.642 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:47 smithi016 ceph-mon[29153]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:21:47.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:47 smithi038 ceph-mon[32841]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:47.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:47 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:47.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:47 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:47.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:47 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:47.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:47 smithi038 ceph-mon[32841]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:21:47.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:47 smithi156 ceph-mon[32631]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:47.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:47 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:21:47.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:47 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:21:47.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:47 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:47.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:47 smithi156 ceph-mon[32631]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:21:48.682 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:48 smithi016 ceph-mon[29153]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:48.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:48 smithi038 ceph-mon[32841]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:48.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:48 smithi156 ceph-mon[32631]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:21:49.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:49 smithi038 ceph-mon[32841]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:49.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:49 smithi156 ceph-mon[32631]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:49.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:49 smithi016 ceph-mon[29153]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:21:50.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:50 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/148933694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f933fcbe-f953-4fb0-8e2f-5eec380563bd"}]: dispatch 2024-04-25T01:21:50.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:50 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/148933694' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f933fcbe-f953-4fb0-8e2f-5eec380563bd"}]': finished 2024-04-25T01:21:50.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:50 smithi016 ceph-mon[29153]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:21:50.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:21:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:50 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/148933694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f933fcbe-f953-4fb0-8e2f-5eec380563bd"}]: dispatch 2024-04-25T01:21:50.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:50 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/148933694' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f933fcbe-f953-4fb0-8e2f-5eec380563bd"}]': finished 2024-04-25T01:21:50.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:50 smithi038 ceph-mon[32841]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:21:50.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:21:50.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:50 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/148933694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f933fcbe-f953-4fb0-8e2f-5eec380563bd"}]: dispatch 2024-04-25T01:21:50.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:50 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/148933694' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f933fcbe-f953-4fb0-8e2f-5eec380563bd"}]': finished 2024-04-25T01:21:50.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:50 smithi156 ceph-mon[32631]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:21:50.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:21:51.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:51 smithi038 ceph-mon[32841]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-25T01:21:51.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:51 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/717979773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:51.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:51 smithi156 ceph-mon[32631]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-25T01:21:51.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:51 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/717979773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:51 smithi016 ceph-mon[29153]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-25T01:21:51.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:51 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/717979773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:21:53.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:53 smithi038 ceph-mon[32841]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:21:53.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:53 smithi156 ceph-mon[32631]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:21:53.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:53 smithi016 ceph-mon[29153]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:21:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:55 smithi016 ceph-mon[29153]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T01:21:55.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:55 smithi038 ceph-mon[32841]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T01:21:55.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:55 smithi156 ceph-mon[32631]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:57 smithi016 ceph-mon[29153]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:57 smithi038 ceph-mon[32841]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T01:21:57.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:57 smithi156 ceph-mon[32631]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T01:21:58.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:58 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:21:58.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:58 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:58.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:58 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:21:58.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:58 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:58.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:58 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:21:58.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:58 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:21:59.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:59 smithi016 ceph-mon[29153]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:59.672 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:21:59 smithi016 ceph-mon[29153]: Deploying daemon osd.6 on smithi016 2024-04-25T01:21:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:59 smithi038 ceph-mon[32841]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:21:59 smithi038 ceph-mon[32841]: Deploying daemon osd.6 on smithi016 2024-04-25T01:21:59.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:59 smithi156 ceph-mon[32631]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:21:59.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:21:59 smithi156 ceph-mon[32631]: Deploying daemon osd.6 on smithi016 2024-04-25T01:22:01.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:01 smithi016 ceph-mon[29153]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:01.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:01 smithi038 ceph-mon[32841]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:01.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:01 smithi156 ceph-mon[32631]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:03.700 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:03 smithi016 ceph-mon[29153]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:03.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:03 smithi038 ceph-mon[32841]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:03.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:03 smithi156 ceph-mon[32631]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:04.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:04 smithi038 ceph-mon[32841]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:04.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:04.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:04.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:05.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:04 smithi156 ceph-mon[32631]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:05.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:05.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:05.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:05.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:04 smithi016 ceph-mon[29153]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:05.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:05.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:05.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:06 smithi038 ceph-mon[32841]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:06.999 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:06 smithi016 ceph-mon[29153]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:07.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:06 smithi156 ceph-mon[32631]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:07.451 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-04-25T01:22:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.469+0000 7f5109ffb700 1 -- 172.21.15.16:0/2578234291 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f510c062670 con 0x7f50fc0a2ca0 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f50fc0a2ca0 msgr2=0x7f50fc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f50fc0a2ca0 0x7f50fc0a5160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f510c06ba40 tx=0x7f50f8005fd0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 msgr2=0x7f510c06b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 0x7f510c06b4c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f510000d980 tx=0x7f510000dc90 comp rx=0 tx=0).stop 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 shutdown_connections 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f50fc0a2ca0 0x7f50fc0a5160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f510c102d50 0x7f510c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f510c1011b0 0x7f510c06b4c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 --2- 172.21.15.16:0/2578234291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c1023b0 0x7f510c19d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.471+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 >> 172.21.15.16:0/2578234291 conn(0x7f510c0fc750 msgr2=0x7f510c1056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.472+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 shutdown_connections 2024-04-25T01:22:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:22:07.472+0000 7f5112cc3700 1 -- 172.21.15.16:0/2578234291 wait complete. 2024-04-25T01:22:08.092 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T01:22:08.094 INFO:tasks.cephadm:Deploying osd.7 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-25T01:22:08.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:22:08.353 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:22:08.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:08 smithi038 ceph-mon[32841]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:22:08.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:08 smithi156 ceph-mon[32631]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:22:08.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:08 smithi016 ceph-mon[29153]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:22:08.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:08.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:09 smithi038 ceph-mon[32841]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:09 smithi038 ceph-mon[32841]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:22:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:09 smithi038 ceph-mon[32841]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:22:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:09 smithi038 ceph-mon[32841]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:22:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:09 smithi156 ceph-mon[32631]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:09 smithi156 ceph-mon[32631]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:22:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:09 smithi156 ceph-mon[32631]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:22:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:09 smithi156 ceph-mon[32631]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:22:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-mon[29153]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-mon[29153]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:22:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-mon[29153]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:22:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-mon[29153]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:22:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:09.862 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:22:09.483+0000 7fdd80dfb700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T01:22:09.863 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:22:09 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:22:09.492+0000 7fdd7cbee700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:22:10.682 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:22:10.682 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:22:10.682 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-25T01:22:10.682 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-25T01:22:10.682 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143634 s, 730 MB/s 2024-04-25T01:22:10.682 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-25T01:22:10.733 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:10 smithi038 ceph-mon[32841]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:22:10.733 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:10 smithi038 ceph-mon[32841]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:22:10.733 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:10 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:10.733 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:10 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:10.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:10 smithi156 ceph-mon[32631]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:22:10.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:10 smithi156 ceph-mon[32631]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:22:10.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:10 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:10.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:10 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:10.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:10 smithi016 ceph-mon[29153]: from='osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:22:10.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:10 smithi016 ceph-mon[29153]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:22:10.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:10 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:10.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:10 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:11.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-25T01:22:11.700 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:22:11.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:22:11.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:22:11.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:11.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:11.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] boot 2024-04-25T01:22:11.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:22:11.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] boot 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:22:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:22:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:22:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:22:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:11.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: osd.6 [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] boot 2024-04-25T01:22:11.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:22:11.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:22:12.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:22:12.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:12.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:12.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:12.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.887 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:12.888 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:13.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 -- 172.21.15.38:0/990398166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 msgr2=0x7fe25010a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 --2- 172.21.15.38:0/990398166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25010a700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe24800b0d0 tx=0x7fe24800b3e0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 -- 172.21.15.38:0/990398166 shutdown_connections 2024-04-25T01:22:13.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 --2- 172.21.15.38:0/990398166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25010a700 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 --2- 172.21.15.38:0/990398166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2501022b0 0x7fe2501026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 --2- 172.21.15.38:0/990398166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2501034b0 0x7fe250103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 -- 172.21.15.38:0/990398166 >> 172.21.15.38:0/990398166 conn(0x7fe2500fd850 msgr2=0x7fe2500ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:13.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.450+0000 7fe257a6e700 1 -- 172.21.15.38:0/990398166 shutdown_connections 2024-04-25T01:22:13.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 -- 172.21.15.38:0/990398166 wait complete. 2024-04-25T01:22:13.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 Processor -- start 2024-04-25T01:22:13.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 -- start start 2024-04-25T01:22:13.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2501022b0 0x7fe250198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2501034b0 0x7fe2501988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe25010dda0 con 0x7fe2501034b0 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe25010dc20 con 0x7fe250103e50 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe257a6e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe25010df20 con 0x7fe2501022b0 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe25600b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe25600b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25019f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52684/0 (socket says 172.21.15.38:52684) 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.451+0000 7fe25600b700 1 -- 172.21.15.38:0/3888211040 learned_addr learned my addr 172.21.15.38:0/3888211040 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25580a700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2501022b0 0x7fe250198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe255009700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2501034b0 0x7fe2501988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25600b700 1 -- 172.21.15.38:0/3888211040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2501022b0 msgr2=0x7fe250198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25600b700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2501022b0 0x7fe250198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25600b700 1 -- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2501034b0 msgr2=0x7fe2501988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:13.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25600b700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2501034b0 0x7fe2501988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25600b700 1 -- 172.21.15.38:0/3888211040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25019fff0 con 0x7fe250103e50 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe25600b700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25019f950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24801f030 con 0x7fe250103e50 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe248003a00 con 0x7fe250103e50 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2501a0280 con 0x7fe250103e50 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2480047f0 con 0x7fe250103e50 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.452+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2501a0750 con 0x7fe250103e50 2024-04-25T01:22:13.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.453+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe250068f20 con 0x7fe250103e50 2024-04-25T01:22:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.454+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe248003bb0 con 0x7fe250103e50 2024-04-25T01:22:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.455+0000 7fe246ffd700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe23c0a2c20 0x7fe23c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.455+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fe24800f020 con 0x7fe250103e50 2024-04-25T01:22:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.455+0000 7fe25580a700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe23c0a2c20 0x7fe23c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.456+0000 7fe25580a700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe23c0a2c20 0x7fe23c0a50e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe250103310 tx=0x7fe24c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.459+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe248029020 con 0x7fe250103e50 2024-04-25T01:22:13.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:13.676+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe250062670 con 0x7fe23c0a2c20 2024-04-25T01:22:13.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:13 smithi156 ceph-mon[32631]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:13.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:13 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi016 2024-04-25T01:22:13.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:13 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi016 to 5697M 2024-04-25T01:22:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:13 smithi016 ceph-mon[29153]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:13.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:13 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi016 2024-04-25T01:22:13.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:13 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi016 to 5697M 2024-04-25T01:22:13.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:13 smithi038 ceph-mon[32841]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:13.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:13 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi016 2024-04-25T01:22:13.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:13 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi016 to 5697M 2024-04-25T01:22:14.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:22:14.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:22:14.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:14.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:22:14.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:22:14.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:14.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:22:14.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:22:14.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:15.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:15 smithi038 ceph-mon[32841]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:15.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:15 smithi038 ceph-mon[32841]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:15.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:15 smithi156 ceph-mon[32631]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:15.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:15 smithi156 ceph-mon[32631]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:15 smithi016 ceph-mon[29153]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:15.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:15 smithi016 ceph-mon[29153]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:16.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:16 smithi156 ceph-mon[32631]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:16.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:16 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/2044607882' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]: dispatch 2024-04-25T01:22:16.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:16 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]: dispatch 2024-04-25T01:22:16.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:16 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]': finished 2024-04-25T01:22:16.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:16 smithi156 ceph-mon[32631]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:22:16.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:16 smithi016 ceph-mon[29153]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:16 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/2044607882' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]: dispatch 2024-04-25T01:22:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:16 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]: dispatch 2024-04-25T01:22:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:16 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]': finished 2024-04-25T01:22:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:16 smithi016 ceph-mon[29153]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:22:16.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:16.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:16 smithi038 ceph-mon[32841]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:16.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:16 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/2044607882' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]: dispatch 2024-04-25T01:22:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:16 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]: dispatch 2024-04-25T01:22:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:16 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da57b8de-edaf-497f-afdb-0d5648bf0c3c"}]': finished 2024-04-25T01:22:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:16 smithi038 ceph-mon[32841]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:22:16.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:17.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/1235594027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:22:17.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/1235594027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:22:17.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/1235594027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:22:18.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:18 smithi016 ceph-mon[29153]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:18 smithi038 ceph-mon[32841]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:19.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:18 smithi156 ceph-mon[32631]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:20.658 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:20 smithi038 ceph-mon[32841]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:20 smithi016 ceph-mon[29153]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:21.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:20 smithi156 ceph-mon[32631]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:22.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:22 smithi016 ceph-mon[29153]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:22.886 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:22 smithi038 ceph-mon[32841]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:23.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:22 smithi156 ceph-mon[32631]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:24.697 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:24 smithi038 ceph-mon[32841]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:24.697 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:22:24.697 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:24 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:24.697 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:24 smithi038 ceph-mon[32841]: Deploying daemon osd.7 on smithi038 2024-04-25T01:22:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:24 smithi156 ceph-mon[32631]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:22:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:24 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:25.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:24 smithi156 ceph-mon[32631]: Deploying daemon osd.7 on smithi038 2024-04-25T01:22:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:24 smithi016 ceph-mon[29153]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:22:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:24 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:24 smithi016 ceph-mon[29153]: Deploying daemon osd.7 on smithi038 2024-04-25T01:22:26.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:26 smithi038 ceph-mon[32841]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:27.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:26 smithi156 ceph-mon[32631]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:27.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:26 smithi016 ceph-mon[29153]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:28.698 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:28 smithi038 ceph-mon[32841]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:29.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:28 smithi156 ceph-mon[32631]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:29.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:28 smithi016 ceph-mon[29153]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:30.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:30 smithi038 ceph-mon[32841]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:30.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:31.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:30 smithi156 ceph-mon[32631]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:31.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:31.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:31.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:30 smithi016 ceph-mon[29153]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:32.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:32 smithi038 ceph-mon[32841]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:33.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:32 smithi156 ceph-mon[32631]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:33.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:32 smithi016 ceph-mon[29153]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.543+0000 7fe246ffd700 1 -- 172.21.15.38:0/3888211040 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe250062670 con 0x7fe23c0a2c20 2024-04-25T01:22:33.525 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 7 on host 'smithi038' 2024-04-25T01:22:33.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe23c0a2c20 msgr2=0x7fe23c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe23c0a2c20 0x7fe23c0a50e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe250103310 tx=0x7fe24c00a380 comp rx=0 tx=0).stop 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 msgr2=0x7fe25019f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25019f950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248004e40 comp rx=0 tx=0).stop 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 shutdown_connections 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe23c0a2c20 0x7fe23c0a50e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe250103e50 0x7fe25019f950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2501022b0 0x7fe250198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 --2- 172.21.15.38:0/3888211040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2501034b0 0x7fe2501988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 >> 172.21.15.38:0/3888211040 conn(0x7fe2500fd850 msgr2=0x7fe2501067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 shutdown_connections 2024-04-25T01:22:33.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:22:33.546+0000 7fe257a6e700 1 -- 172.21.15.38:0/3888211040 wait complete. 2024-04-25T01:22:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:34.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:22:34.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:22:34.168 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:34.193 DEBUG:teuthology.orchestra.run.smithi038:osd.7> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T01:22:34.196 INFO:tasks.cephadm:Deploying osd.8 on smithi156 with /dev/vg_nvme/lv_2... 2024-04-25T01:22:34.196 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:22:34.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:34.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:34.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:34.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:22:34.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:34.444 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:22:34.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:34 smithi156 ceph-mon[32631]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:34.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:34 smithi156 ceph-mon[32631]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:22:34.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:34 smithi156 ceph-mon[32631]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:22:34.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:34 smithi156 ceph-mon[32631]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:22:34.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:34 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:35.109 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:34 smithi038 ceph-mon[32841]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:35.109 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:34 smithi038 ceph-mon[32841]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:22:35.109 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:34 smithi038 ceph-mon[32841]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:22:35.109 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:34 smithi038 ceph-mon[32841]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:22:35.109 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:34 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:34 smithi016 ceph-mon[29153]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:34 smithi016 ceph-mon[29153]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:22:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:34 smithi016 ceph-mon[29153]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:22:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:34 smithi016 ceph-mon[29153]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:22:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:34 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:35.717 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:22:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:22:35.558+0000 7f428b823700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:22:35.718 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:22:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:22:35.570+0000 7f4287616700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:22:36.480 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00859645 s, 1.2 GB/s 2024-04-25T01:22:36.481 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T01:22:36.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:36 smithi156 ceph-mon[32631]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:22:36.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:36 smithi156 ceph-mon[32631]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:22:36.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:36.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:36.828 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:36 smithi156 ceph-mon[32631]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:36 smithi016 ceph-mon[29153]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:22:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:36 smithi016 ceph-mon[29153]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:22:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:36 smithi016 ceph-mon[29153]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:36.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:36 smithi038 ceph-mon[32841]: from='osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:22:36.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:36 smithi038 ceph-mon[32841]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:22:36.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:36.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:36.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:36 smithi038 ceph-mon[32841]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:22:37.154 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2024-04-25T01:22:37.441 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:22:37.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:22:37.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] boot 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi038 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:37.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:37.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.797 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] boot 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi038 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-25T01:22:37.798 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.799 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:37.799 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:37.799 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: osd.7 [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] boot 2024-04-25T01:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: osdmap e51: 8 total, 8 up, 8 in 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi038 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:38.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:38 smithi156 ceph-mon[32631]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:22:38.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:38 smithi156 ceph-mon[32631]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:38 smithi016 ceph-mon[29153]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:22:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:38 smithi016 ceph-mon[29153]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:38.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:38 smithi038 ceph-mon[32841]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:22:38.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:38 smithi038 ceph-mon[32841]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:39.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.093+0000 7f8998222700 1 -- 172.21.15.156:0/2730565442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990102380 msgr2=0x7f89901027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.093+0000 7f8998222700 1 --2- 172.21.15.156:0/2730565442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990102380 0x7f89901027e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8980009990 tx=0x7f8980009ca0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- 172.21.15.156:0/2730565442 shutdown_connections 2024-04-25T01:22:39.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 --2- 172.21.15.156:0/2730565442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102d20 0x7f89901095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 --2- 172.21.15.156:0/2730565442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990101180 0x7f89901015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 --2- 172.21.15.156:0/2730565442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990102380 0x7f89901027e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- 172.21.15.156:0/2730565442 >> 172.21.15.156:0/2730565442 conn(0x7f89900fc720 msgr2=0x7f89900feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- 172.21.15.156:0/2730565442 shutdown_connections 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- 172.21.15.156:0/2730565442 wait complete. 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 Processor -- start 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- start start 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 0x7f8990105110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.078 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102380 0x7f8990103760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990102d20 0x7f8990103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f899010cd10 con 0x7f8990102d20 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f899010cb90 con 0x7f8990102380 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8998222700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f899010ce90 con 0x7f8990101180 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.094+0000 7f8995fbe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 0x7f8990105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 0x7f8990105110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:44674/0 (socket says 172.21.15.156:44674) 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 -- 172.21.15.156:0/2765520991 learned_addr learned my addr 172.21.15.156:0/2765520991 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f89957bd700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102380 0x7f8990103760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f89967bf700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990102d20 0x7f8990103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 -- 172.21.15.156:0/2765520991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102380 msgr2=0x7f8990103760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102380 0x7f8990103760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 -- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990102d20 msgr2=0x7f8990103ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:39.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990102d20 0x7f8990103ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 -- 172.21.15.156:0/2765520991 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8990104550 con 0x7f8990101180 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f89957bd700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102380 0x7f8990103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f89967bf700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990102d20 0x7f8990103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8995fbe700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 0x7f8990105110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f898c00b500 tx=0x7f898c00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c015020 con 0x7f8990101180 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f898c011ce0 con 0x7f8990101180 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c010d60 con 0x7f8990101180 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89901a2710 con 0x7f8990101180 2024-04-25T01:22:39.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.095+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89901a2ca0 con 0x7f8990101180 2024-04-25T01:22:39.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.096+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8990068f20 con 0x7f8990101180 2024-04-25T01:22:39.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.097+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f898c018020 con 0x7f8990101180 2024-04-25T01:22:39.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.098+0000 7f8986ffd700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f897c0a2c20 0x7f897c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:39.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.098+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f898c0ce2b0 con 0x7f8990101180 2024-04-25T01:22:39.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.098+0000 7f89957bd700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f897c0a2c20 0x7f897c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:39.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.099+0000 7f89957bd700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f897c0a2c20 0x7f897c0a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f898000b5c0 tx=0x7f8980005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:39.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.102+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f898c01f020 con 0x7f8990101180 2024-04-25T01:22:39.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:39.311+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f899006b100 con 0x7f897c0a2c20 2024-04-25T01:22:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:39 smithi016 ceph-mon[29153]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:22:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:39 smithi016 ceph-mon[29153]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:22:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:22:39.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:39.904 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:39 smithi156 ceph-mon[32631]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:22:39.904 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:39 smithi156 ceph-mon[32631]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:22:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:22:39.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:39.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:39 smithi038 ceph-mon[32841]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:22:39.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:39 smithi038 ceph-mon[32841]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:39.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:22:39.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:22:39.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:40 smithi016 ceph-mon[29153]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:40 smithi016 ceph-mon[29153]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:22:40.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:40 smithi156 ceph-mon[32631]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:40.887 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:40 smithi156 ceph-mon[32631]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:22:40.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:40 smithi038 ceph-mon[32841]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:40.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:40 smithi038 ceph-mon[32841]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:22:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:41 smithi016 ceph-mon[29153]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:22:41.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:41 smithi038 ceph-mon[32841]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:22:42.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:41 smithi156 ceph-mon[32631]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:22:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:42 smithi016 ceph-mon[29153]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:42 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/4008132065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]: dispatch 2024-04-25T01:22:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:42 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]: dispatch 2024-04-25T01:22:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:42 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]': finished 2024-04-25T01:22:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:42 smithi016 ceph-mon[29153]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T01:22:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:22:42.868 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:42 smithi038 ceph-mon[32841]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:42.868 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:42 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/4008132065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]: dispatch 2024-04-25T01:22:42.869 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:42 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]: dispatch 2024-04-25T01:22:42.869 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:42 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]': finished 2024-04-25T01:22:42.869 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:42 smithi038 ceph-mon[32841]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T01:22:42.869 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:22:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:42 smithi156 ceph-mon[32631]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:42 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/4008132065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]: dispatch 2024-04-25T01:22:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:42 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]: dispatch 2024-04-25T01:22:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:42 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6850fc5-5935-403c-bbe2-c2fc2e73d3cd"}]': finished 2024-04-25T01:22:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:42 smithi156 ceph-mon[32631]: osdmap e56: 9 total, 8 up, 9 in 2024-04-25T01:22:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:22:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:43 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/3274829115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:22:43.884 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:43 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/3274829115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:22:44.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:43 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/3274829115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:22:44.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:44 smithi016 ceph-mon[29153]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:44.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:44 smithi038 ceph-mon[32841]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:45.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:44 smithi156 ceph-mon[32631]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:46.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:46 smithi156 ceph-mon[32631]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:46 smithi016 ceph-mon[29153]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:46.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:46 smithi038 ceph-mon[32841]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:48.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:48 smithi156 ceph-mon[32631]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:48.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:48 smithi016 ceph-mon[29153]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:48 smithi038 ceph-mon[32841]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:50.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:50 smithi016 ceph-mon[29153]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:50.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:22:50.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:50.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:50 smithi016 ceph-mon[29153]: Deploying daemon osd.8 on smithi156 2024-04-25T01:22:50.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:50 smithi038 ceph-mon[32841]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:50.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:22:50.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:50.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:50 smithi038 ceph-mon[32841]: Deploying daemon osd.8 on smithi156 2024-04-25T01:22:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:50 smithi156 ceph-mon[32631]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:22:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:50 smithi156 ceph-mon[32631]: Deploying daemon osd.8 on smithi156 2024-04-25T01:22:52.817 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:52 smithi156 ceph-mon[32631]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:52.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:52 smithi016 ceph-mon[29153]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:52.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:52 smithi038 ceph-mon[32841]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:54.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:54 smithi016 ceph-mon[29153]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:54.926 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:54 smithi156 ceph-mon[32631]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:54.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:54 smithi038 ceph-mon[32841]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:56.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:56.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:56.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:56.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:56 smithi156 ceph-mon[32631]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:56 smithi016 ceph-mon[29153]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:56.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:56.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:56.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:56.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:56 smithi038 ceph-mon[32841]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:58.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:58 smithi038 ceph-mon[32841]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:59.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:58 smithi156 ceph-mon[32631]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:59.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:58 smithi016 ceph-mon[29153]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:22:59.474 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 8 on host 'smithi156' 2024-04-25T01:22:59.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.490+0000 7f8986ffd700 1 -- 172.21.15.156:0/2765520991 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f899006b100 con 0x7f897c0a2c20 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f897c0a2c20 msgr2=0x7f897c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f897c0a2c20 0x7f897c0a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f898000b5c0 tx=0x7f8980005fb0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 msgr2=0x7f8990105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 0x7f8990105110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f898c00b500 tx=0x7f898c00b8c0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 shutdown_connections 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f897c0a2c20 0x7f897c0a50e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990102d20 0x7f8990103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8990101180 0x7f8990105110 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 --2- 172.21.15.156:0/2765520991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8990102380 0x7f8990103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.492+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 >> 172.21.15.156:0/2765520991 conn(0x7f89900fc720 msgr2=0x7f899006a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:59.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.493+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 shutdown_connections 2024-04-25T01:22:59.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:22:59.493+0000 7f8998222700 1 -- 172.21.15.156:0/2765520991 wait complete. 2024-04-25T01:22:59.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:22:59.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:22:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:22:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:22:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:00.122 DEBUG:teuthology.orchestra.run.smithi156:osd.8> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T01:23:00.124 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-04-25T01:23:00.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:23:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:00.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-mon[32631]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:23:00.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:23:00.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-mon[32631]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:23:00.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:00.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-mon[32631]: from='osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:23:00.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:23:00.752 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:23:00.666+0000 7fbe2fc44700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T01:23:00.753 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:23:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:23:00.675+0000 7fbe2ba37700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:23:00.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:00 smithi016 ceph-mon[29153]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:23:00.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:00 smithi016 ceph-mon[29153]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:23:00.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:00 smithi016 ceph-mon[29153]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:23:00.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:00.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:00 smithi016 ceph-mon[29153]: from='osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:23:00.756 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:00 smithi016 ceph-mon[29153]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:23:00.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:00 smithi038 ceph-mon[32841]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:23:00.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:00 smithi038 ceph-mon[32841]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:23:00.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:00 smithi038 ceph-mon[32841]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:23:00.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:00.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:00 smithi038 ceph-mon[32841]: from='osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:23:00.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:00 smithi038 ceph-mon[32841]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:23:01.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:23:01.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:23:01.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] boot 2024-04-25T01:23:01.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T01:23:01.823 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.872 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:23:01.873 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:23:01.873 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.873 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.873 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] boot 2024-04-25T01:23:01.873 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T01:23:01.873 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:23:01.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:23:01.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:01.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: osd.8 [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] boot 2024-04-25T01:23:01.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: osdmap e59: 9 total, 9 up, 9 in 2024-04-25T01:23:01.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:23:02.435 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:23:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:23:02.436 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-25T01:23:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-25T01:23:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00857614 s, 1.2 GB/s 2024-04-25T01:23:02.436 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-25T01:23:02.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:02 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:23:02.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:02 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:23:02.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:02 smithi156 ceph-mon[32631]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:23:02.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:02 smithi156 ceph-mon[32631]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T01:23:02.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:02 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:23:02.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:02 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:23:02.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:02 smithi038 ceph-mon[32841]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:23:02.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:02 smithi038 ceph-mon[32841]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T01:23:03.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:02 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:23:03.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:02 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:23:03.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:02 smithi016 ceph-mon[29153]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:23:03.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:02 smithi016 ceph-mon[29153]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T01:23:03.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-04-25T01:23:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi156 2024-04-25T01:23:04.670 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.670 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.670 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.670 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:04 smithi016 ceph-mon[29153]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi156 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-25T01:23:04.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:04.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:04.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:04 smithi038 ceph-mon[32841]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi156 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:04.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:04.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:04.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:04 smithi156 ceph-mon[32631]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:23:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 -- 172.21.15.16:0/1773522005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103e30 msgr2=0x7f790c10a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 --2- 172.21.15.16:0/1773522005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103e30 0x7f790c10a6e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f790400b0d0 tx=0x7f790400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 -- 172.21.15.16:0/1773522005 shutdown_connections 2024-04-25T01:23:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 --2- 172.21.15.16:0/1773522005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103e30 0x7f790c10a6e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 --2- 172.21.15.16:0/1773522005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f790c102290 0x7f790c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 --2- 172.21.15.16:0/1773522005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f790c103490 0x7f790c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 -- 172.21.15.16:0/1773522005 >> 172.21.15.16:0/1773522005 conn(0x7f790c0fd830 msgr2=0x7f790c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 -- 172.21.15.16:0/1773522005 shutdown_connections 2024-04-25T01:23:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.332+0000 7f7912e4c700 1 -- 172.21.15.16:0/1773522005 wait complete. 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 Processor -- start 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 -- start start 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f790c102290 0x7f790c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 0x7f790c198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f790c103e30 0x7f790c19f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f790c10dd20 con 0x7f790c103490 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f790c10dba0 con 0x7f790c103e30 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7912e4c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f790c10dea0 con 0x7f790c102290 2024-04-25T01:23:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 0x7f790c198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 0x7f790c198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52968/0 (socket says 172.21.15.16:52968) 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 -- 172.21.15.16:0/3118045559 learned_addr learned my addr 172.21.15.16:0/3118045559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f7910be8700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f790c102290 0x7f790c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f79113e9700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f790c103e30 0x7f790c19f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 -- 172.21.15.16:0/3118045559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f790c102290 msgr2=0x7f790c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f790c102290 0x7f790c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 -- 172.21.15.16:0/3118045559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f790c103e30 msgr2=0x7f790c19f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f790c103e30 0x7f790c19f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.333+0000 7f790bfff700 1 -- 172.21.15.16:0/3118045559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c19ffa0 con 0x7f790c103490 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.334+0000 7f790bfff700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 0x7f790c198880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78f800e9f0 tx=0x7f78f800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.334+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f800cbd0 con 0x7f790c103490 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.334+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78f800ee90 con 0x7f790c103490 2024-04-25T01:23:05.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.334+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790c1a0290 con 0x7f790c103490 2024-04-25T01:23:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.334+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8005590 con 0x7f790c103490 2024-04-25T01:23:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.334+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f790c1a07d0 con 0x7f790c103490 2024-04-25T01:23:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.335+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f790c068f20 con 0x7f790c103490 2024-04-25T01:23:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.336+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f78f8004750 con 0x7f790c103490 2024-04-25T01:23:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.337+0000 7f7909ffb700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78fc0a2ca0 0x7f78fc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.337+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f78f80cea90 con 0x7f790c103490 2024-04-25T01:23:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.337+0000 7f7910be8700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78fc0a2ca0 0x7f78fc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.337+0000 7f7910be8700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78fc0a2ca0 0x7f78fc0a5160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f790c1032f0 tx=0x7f7900009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.340+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f78f8053740 con 0x7f790c103490 2024-04-25T01:23:05.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:05 smithi016 ceph-mon[29153]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:05.571+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f790c062670 con 0x7f78fc0a2ca0 2024-04-25T01:23:05.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:05 smithi038 ceph-mon[32841]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:05.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:05 smithi156 ceph-mon[32631]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:06.543 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:06 smithi016 ceph-mon[29153]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:23:06.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:06.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:06.544 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:06.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:06 smithi038 ceph-mon[32841]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:23:06.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:06.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:06.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:06.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:06 smithi156 ceph-mon[32631]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:23:06.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:06.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:06.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:07 smithi016 ceph-mon[29153]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:07 smithi016 ceph-mon[29153]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:07 smithi016 ceph-mon[29153]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:23:07.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:07 smithi038 ceph-mon[32841]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:07.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:07 smithi038 ceph-mon[32841]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:07.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:07 smithi038 ceph-mon[32841]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:23:07.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:07 smithi156 ceph-mon[32631]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:07.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:07 smithi156 ceph-mon[32631]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:07.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:07 smithi156 ceph-mon[32631]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:23:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:09 smithi016 ceph-mon[29153]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-25T01:23:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:09 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1718965769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d90517c5-56a1-45f6-adf5-6c2b28702bc2"}]: dispatch 2024-04-25T01:23:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:09 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1718965769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d90517c5-56a1-45f6-adf5-6c2b28702bc2"}]': finished 2024-04-25T01:23:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:09 smithi016 ceph-mon[29153]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T01:23:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:09 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/384420838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:23:09.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:09 smithi038 ceph-mon[32841]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-25T01:23:09.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:09 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1718965769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d90517c5-56a1-45f6-adf5-6c2b28702bc2"}]: dispatch 2024-04-25T01:23:09.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:09 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1718965769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d90517c5-56a1-45f6-adf5-6c2b28702bc2"}]': finished 2024-04-25T01:23:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:09 smithi038 ceph-mon[32841]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T01:23:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:09.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:09 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/384420838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:23:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:09 smithi156 ceph-mon[32631]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-25T01:23:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:09 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1718965769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d90517c5-56a1-45f6-adf5-6c2b28702bc2"}]: dispatch 2024-04-25T01:23:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:09 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1718965769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d90517c5-56a1-45f6-adf5-6c2b28702bc2"}]': finished 2024-04-25T01:23:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:09 smithi156 ceph-mon[32631]: osdmap e64: 10 total, 9 up, 10 in 2024-04-25T01:23:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:09.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:09 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/384420838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:23:11.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:11 smithi156 ceph-mon[32631]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T01:23:11.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:11 smithi016 ceph-mon[29153]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T01:23:11.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:11 smithi038 ceph-mon[32841]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T01:23:13.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:13 smithi016 ceph-mon[29153]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-25T01:23:13.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:13 smithi038 ceph-mon[32841]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-25T01:23:13.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:13 smithi156 ceph-mon[32631]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-25T01:23:15.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:15 smithi016 ceph-mon[29153]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:23:15.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:15 smithi038 ceph-mon[32841]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:23:15.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:15 smithi156 ceph-mon[32631]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:23:16.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:23:16.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:16.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:23:16.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:16.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:23:16.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:17.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:17 smithi016 ceph-mon[29153]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-25T01:23:17.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:17 smithi016 ceph-mon[29153]: Deploying daemon osd.9 on smithi016 2024-04-25T01:23:17.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:17 smithi038 ceph-mon[32841]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-25T01:23:17.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:17 smithi038 ceph-mon[32841]: Deploying daemon osd.9 on smithi016 2024-04-25T01:23:17.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:17 smithi156 ceph-mon[32631]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-25T01:23:17.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:17 smithi156 ceph-mon[32631]: Deploying daemon osd.9 on smithi016 2024-04-25T01:23:19.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:19 smithi016 ceph-mon[29153]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:19.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:19 smithi038 ceph-mon[32841]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:19.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:19 smithi156 ceph-mon[32631]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:21.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:21 smithi016 ceph-mon[29153]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:21.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:21 smithi038 ceph-mon[32841]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:21 smithi156 ceph-mon[32631]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:23.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:23 smithi016 ceph-mon[29153]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:23.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:23.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:23.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:23 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:23.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:23 smithi156 ceph-mon[32631]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:23.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:23.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:23.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:23 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:23.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:23 smithi038 ceph-mon[32841]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:23.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:23.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:23.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:23 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:25 smithi016 ceph-mon[29153]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:25.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:25 smithi038 ceph-mon[32841]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:25.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:25 smithi156 ceph-mon[32631]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.047+0000 7f7909ffb700 1 -- 172.21.15.16:0/3118045559 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f790c062670 con 0x7f78fc0a2ca0 2024-04-25T01:23:26.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78fc0a2ca0 msgr2=0x7f78fc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78fc0a2ca0 0x7f78fc0a5160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f790c1032f0 tx=0x7f7900009260 comp rx=0 tx=0).stop 2024-04-25T01:23:26.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 msgr2=0x7f790c198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 0x7f790c198880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78f800e9f0 tx=0x7f78f800ed00 comp rx=0 tx=0).stop 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 shutdown_connections 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78fc0a2ca0 0x7f78fc0a5160 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f790c103e30 0x7f790c19f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f790c102290 0x7f790c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 --2- 172.21.15.16:0/3118045559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790c103490 0x7f790c198880 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.049+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 >> 172.21.15.16:0/3118045559 conn(0x7f790c0fd830 msgr2=0x7f790c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.050+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 shutdown_connections 2024-04-25T01:23:26.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:23:26.050+0000 7f7912e4c700 1 -- 172.21.15.16:0/3118045559 wait complete. 2024-04-25T01:23:26.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:26 smithi016 ceph-mon[29153]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:23:26.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:26 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:26.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:26 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:26.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:26 smithi038 ceph-mon[32841]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:23:26.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:26 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:26.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:26 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:26.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:26 smithi156 ceph-mon[32631]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:23:26.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:26 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:26.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:26 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:26.856 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T01:23:26.859 INFO:tasks.cephadm:Deploying osd.10 on smithi038 with /dev/vg_nvme/lv_1... 2024-04-25T01:23:26.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:23:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:27.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:27 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:27.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:27.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:23:27.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T01:23:27.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-25T01:23:27.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:27.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:27.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:27 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:27.824 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:23:27.559+0000 7f5006eef700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T01:23:27.824 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:23:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:23:27.576+0000 7f4ffecda700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:23:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:28 smithi016 ceph-mon[29153]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:23:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:28 smithi016 ceph-mon[29153]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:23:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:28 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:28 smithi016 ceph-mon[29153]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' 2024-04-25T01:23:28.399 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:28 smithi038 ceph-mon[32841]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:23:28.399 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:28 smithi038 ceph-mon[32841]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:23:28.399 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:28 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:28.399 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:28 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:28.399 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:28 smithi038 ceph-mon[32841]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' 2024-04-25T01:23:28.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:28 smithi156 ceph-mon[32631]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-25T01:23:28.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:28 smithi156 ceph-mon[32631]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:23:28.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:28 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:28.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:28 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:28.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:28 smithi156 ceph-mon[32631]: from='osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684]' entity='osd.9' 2024-04-25T01:23:29.312 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:23:29.312 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:23:29.313 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:29.313 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:29.313 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] boot 2024-04-25T01:23:29.313 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T01:23:29.313 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:29 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:29.313 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:23:29.313 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:23:29.313 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-25T01:23:29.313 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-25T01:23:29.313 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155489 s, 674 MB/s 2024-04-25T01:23:29.313 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-25T01:23:29.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:23:29.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:23:29.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:29.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:29.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] boot 2024-04-25T01:23:29.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T01:23:29.319 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:29 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:29.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:23:29.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:23:29.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:23:29.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:29.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: osd.9 [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] boot 2024-04-25T01:23:29.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: osdmap e67: 10 total, 10 up, 10 in 2024-04-25T01:23:29.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:29 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:23:30.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-04-25T01:23:30.290 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:23:30.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:30 smithi016 ceph-mon[29153]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T01:23:30.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:30 smithi016 ceph-mon[29153]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:30.703 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:30 smithi038 ceph-mon[32841]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T01:23:30.703 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:30 smithi038 ceph-mon[32841]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:31.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:30 smithi156 ceph-mon[32631]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T01:23:31.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:30 smithi156 ceph-mon[32631]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:31.835 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi016 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:31.836 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:31.837 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:31.837 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:31 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:31.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 -- 172.21.15.38:0/1348415437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 msgr2=0x7f39181026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 --2- 172.21.15.38:0/1348415437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f39181026e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f391000b0d0 tx=0x7f391000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 -- 172.21.15.38:0/1348415437 shutdown_connections 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 --2- 172.21.15.38:0/1348415437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3918103e60 0x7f391810a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 --2- 172.21.15.38:0/1348415437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f39181026e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 --2- 172.21.15.38:0/1348415437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39181034c0 0x7f3918103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 -- 172.21.15.38:0/1348415437 >> 172.21.15.38:0/1348415437 conn(0x7f39180fd880 msgr2=0x7f39180ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 -- 172.21.15.38:0/1348415437 shutdown_connections 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.960+0000 7f391d62d700 1 -- 172.21.15.38:0/1348415437 wait complete. 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 Processor -- start 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 -- start start 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f3918075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39181034c0 0x7f3918074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3918103e60 0x7f3918074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f391810dd90 con 0x7f3918103e60 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f391810dc10 con 0x7f39181022c0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f391d62d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f391810df10 con 0x7f39181034c0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f3918075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f3918075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39804/0 (socket says 172.21.15.38:39804) 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 -- 172.21.15.38:0/2611191696 learned_addr learned my addr 172.21.15.38:0/2611191696 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f39167fc700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39181034c0 0x7f3918074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f39177fe700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3918103e60 0x7f3918074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 -- 172.21.15.38:0/2611191696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39181034c0 msgr2=0x7f3918074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39181034c0 0x7f3918074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 -- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3918103e60 msgr2=0x7f3918074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3918103e60 0x7f3918074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f3916ffd700 1 -- 172.21.15.38:0/2611191696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3918075410 con 0x7f39181022c0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.961+0000 7f39177fe700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3918103e60 0x7f3918074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.962+0000 7f3916ffd700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f3918075fd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3910009550 tx=0x7f3910004d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.962+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391001f030 con 0x7f39181022c0 2024-04-25T01:23:31.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.962+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39181a2540 con 0x7f39181022c0 2024-04-25T01:23:31.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.962+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3910005040 con 0x7f39181022c0 2024-04-25T01:23:31.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.962+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391001b690 con 0x7f39181022c0 2024-04-25T01:23:31.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.962+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39181a2990 con 0x7f39181022c0 2024-04-25T01:23:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.963+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3918068f20 con 0x7f39181022c0 2024-04-25T01:23:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.964+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3910022030 con 0x7f39181022c0 2024-04-25T01:23:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.966+0000 7f390ffff700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39040a2c20 0x7f39040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.966+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f391000f020 con 0x7f39181022c0 2024-04-25T01:23:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.966+0000 7f39167fc700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39040a2c20 0x7f39040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.966+0000 7f39167fc700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39040a2c20 0x7f39040a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3900005950 tx=0x7f390000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:31.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:31.971+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3910029020 con 0x7f39181022c0 2024-04-25T01:23:32.013 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi016 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:32.014 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:32.015 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:31 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:32.186+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3918062670 con 0x7f39040a2c20 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi016 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-25T01:23:32.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:32.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:31 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:33.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:33 smithi038 ceph-mon[32841]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:33.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:33 smithi038 ceph-mon[32841]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:33.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:33.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:33.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:33 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:33.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:33 smithi156 ceph-mon[32631]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:33.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:33 smithi156 ceph-mon[32631]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:33.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:33.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:33.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:33 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:33.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:33 smithi016 ceph-mon[29153]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:33.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:33 smithi016 ceph-mon[29153]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:33.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:33.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:33.267 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:33 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:35.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:35 smithi156 ceph-mon[32631]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:35.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:35 smithi016 ceph-mon[29153]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:35.455 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:35 smithi038 ceph-mon[32841]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:36.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:36 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/92993401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]: dispatch 2024-04-25T01:23:36.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:36 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]: dispatch 2024-04-25T01:23:36.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:36 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]': finished 2024-04-25T01:23:36.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:36 smithi156 ceph-mon[32631]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T01:23:36.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:36.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:36 smithi156 ceph-mon[32631]: from='client.? 172.21.15.38:0/3194065123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:23:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:36 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/92993401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]: dispatch 2024-04-25T01:23:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:36 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]: dispatch 2024-04-25T01:23:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:36 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]': finished 2024-04-25T01:23:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:36 smithi016 ceph-mon[29153]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T01:23:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:36 smithi016 ceph-mon[29153]: from='client.? 172.21.15.38:0/3194065123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:23:36.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:36 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/92993401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]: dispatch 2024-04-25T01:23:36.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:36 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]: dispatch 2024-04-25T01:23:36.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:36 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3da7ce51-60f7-45d6-87d3-42baee7a62db"}]': finished 2024-04-25T01:23:36.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:36 smithi038 ceph-mon[32841]: osdmap e69: 11 total, 10 up, 11 in 2024-04-25T01:23:36.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:36.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:36 smithi038 ceph-mon[32841]: from='client.? 172.21.15.38:0/3194065123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:23:37.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:37 smithi016 ceph-mon[29153]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:37.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:37 smithi038 ceph-mon[32841]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:37.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:37 smithi156 ceph-mon[32631]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:39.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:39 smithi016 ceph-mon[29153]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:39.454 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:39 smithi038 ceph-mon[32841]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:39.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:39 smithi156 ceph-mon[32631]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:41.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:41 smithi016 ceph-mon[29153]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:41.419 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:41 smithi038 ceph-mon[32841]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:41.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:41 smithi156 ceph-mon[32631]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:43.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:43 smithi016 ceph-mon[29153]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:43.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:23:43.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:43.399 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:43 smithi038 ceph-mon[32841]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:43.400 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:23:43.400 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:43.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:43 smithi156 ceph-mon[32631]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:43.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:23:43.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:44.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:44 smithi016 ceph-mon[29153]: Deploying daemon osd.10 on smithi038 2024-04-25T01:23:44.433 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:44 smithi038 ceph-mon[32841]: Deploying daemon osd.10 on smithi038 2024-04-25T01:23:44.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:44 smithi156 ceph-mon[32631]: Deploying daemon osd.10 on smithi038 2024-04-25T01:23:45.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:45 smithi016 ceph-mon[29153]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:45.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:45 smithi038 ceph-mon[32841]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:45.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:45 smithi156 ceph-mon[32631]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:47.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:47 smithi038 ceph-mon[32841]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:47.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:47 smithi016 ceph-mon[29153]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:47.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:47 smithi156 ceph-mon[32631]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:49.262 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:49 smithi038 ceph-mon[32841]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:49.262 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:49.262 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:49.262 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:49.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:49 smithi016 ceph-mon[29153]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:49.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:49.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:49.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:49.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:49 smithi156 ceph-mon[32631]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:49.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:49.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:49.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:51.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:51 smithi038 ceph-mon[32841]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:51.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:51 smithi016 ceph-mon[29153]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:51.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:51 smithi156 ceph-mon[32631]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:52.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.656+0000 7f390ffff700 1 -- 172.21.15.38:0/2611191696 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3918062670 con 0x7f39040a2c20 2024-04-25T01:23:52.637 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 10 on host 'smithi038' 2024-04-25T01:23:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39040a2c20 msgr2=0x7f39040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39040a2c20 0x7f39040a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3900005950 tx=0x7f390000a300 comp rx=0 tx=0).stop 2024-04-25T01:23:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 msgr2=0x7f3918075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f3918075fd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3910009550 tx=0x7f3910004d70 comp rx=0 tx=0).stop 2024-04-25T01:23:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 shutdown_connections 2024-04-25T01:23:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39040a2c20 0x7f39040a50e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:52.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3918103e60 0x7f3918074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:52.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39181022c0 0x7f3918075fd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:52.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 --2- 172.21.15.38:0/2611191696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39181034c0 0x7f3918074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:52.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 >> 172.21.15.38:0/2611191696 conn(0x7f39180fd880 msgr2=0x7f3918106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:52.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 shutdown_connections 2024-04-25T01:23:52.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-25T01:23:52.659+0000 7f391d62d700 1 -- 172.21.15.38:0/2611191696 wait complete. 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:52.946 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:23:52.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:23:52.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:52.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:52.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:53.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:53.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:53.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:53.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:23:53.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:23:53.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:53.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:53.393 DEBUG:teuthology.orchestra.run.smithi038:osd.10> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T01:23:53.395 INFO:tasks.cephadm:Deploying osd.11 on smithi156 with /dev/vg_nvme/lv_1... 2024-04-25T01:23:53.395 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:23:53.647 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:23:54.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-mon[32841]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:23:54.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-mon[32841]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T01:23:54.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:54.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-mon[32841]: from='osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:23:54.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-mon[32841]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:23:54.215 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:23:54.150+0000 7f60c2d01700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T01:23:54.215 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:23:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:23:54.163+0000 7f60beaf4700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:23:54.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:54 smithi156 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:23:54.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:54 smithi156 ceph-mon[32631]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T01:23:54.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:54.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:54 smithi156 ceph-mon[32631]: from='osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:23:54.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:54 smithi156 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:23:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:54 smithi016 ceph-mon[29153]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:23:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:54 smithi016 ceph-mon[29153]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T01:23:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:54 smithi016 ceph-mon[29153]: from='osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:23:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:54 smithi016 ceph-mon[29153]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] boot 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T01:23:55.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:55 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] boot 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T01:23:55.393 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:55 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: osd.10 [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] boot 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: osdmap e72: 11 total, 11 up, 11 in 2024-04-25T01:23:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:55 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:23:55.726 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:23:55.726 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:23:55.726 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T01:23:55.727 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T01:23:55.727 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00845229 s, 1.2 GB/s 2024-04-25T01:23:55.727 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T01:23:56.456 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:56 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:23:56.456 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:56 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:23:56.456 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:56 smithi156 ceph-mon[32631]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T01:23:56.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:56 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:23:56.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:56 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:23:56.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:56 smithi038 ceph-mon[32841]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T01:23:56.484 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2024-04-25T01:23:56.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:56 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:23:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:56 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:23:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:56 smithi016 ceph-mon[29153]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T01:23:56.727 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:57.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:57.468 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:23:57.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:23:57.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:57.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:57.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:57.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:23:58.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 -- 172.21.15.156:0/538862934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990102d20 msgr2=0x7f39901095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:58.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 --2- 172.21.15.156:0/538862934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990102d20 0x7f39901095d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f398800b0a0 tx=0x7f398800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:23:58.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 -- 172.21.15.156:0/538862934 shutdown_connections 2024-04-25T01:23:58.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 --2- 172.21.15.156:0/538862934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990102d20 0x7f39901095d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:58.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 --2- 172.21.15.156:0/538862934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3990101180 0x7f39901015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:58.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 --2- 172.21.15.156:0/538862934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3990102380 0x7f39901027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 -- 172.21.15.156:0/538862934 >> 172.21.15.156:0/538862934 conn(0x7f39900fc720 msgr2=0x7f39900feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 -- 172.21.15.156:0/538862934 shutdown_connections 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.433+0000 7f3994c09700 1 -- 172.21.15.156:0/538862934 wait complete. 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 Processor -- start 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 -- start start 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 0x7f399006b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3990102380 0x7f399019d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3990102d20 0x7f399019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f399010ccb0 con 0x7f3990102d20 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f399010cb30 con 0x7f3990102380 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f3994c09700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f399010ce30 con 0x7f3990101180 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f398e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 0x7f399006b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:58.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f398e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 0x7f399006b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56962/0 (socket says 172.21.15.156:56962) 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f398e59c700 1 -- 172.21.15.156:0/2284027169 learned_addr learned my addr 172.21.15.156:0/2284027169 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f398dd9b700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3990102380 0x7f399019d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.434+0000 7f398ed9d700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3990102d20 0x7f399019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f398e59c700 1 -- 172.21.15.156:0/2284027169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3990102380 msgr2=0x7f399019d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f398e59c700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3990102380 0x7f399019d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f398e59c700 1 -- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3990102d20 msgr2=0x7f399019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f398e59c700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3990102d20 0x7f399019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f398e59c700 1 -- 172.21.15.156:0/2284027169 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39901a0050 con 0x7f3990101180 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f398e59c700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 0x7f399006b320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f398000b500 tx=0x7f398000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980015020 con 0x7f3990101180 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3980011ce0 con 0x7f3990101180 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39901a0340 con 0x7f3990101180 2024-04-25T01:23:58.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980010c80 con 0x7f3990101180 2024-04-25T01:23:58.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.435+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39901a0900 con 0x7f3990101180 2024-04-25T01:23:58.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.436+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3990068f20 con 0x7f3990101180 2024-04-25T01:23:58.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.437+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3980018020 con 0x7f3990101180 2024-04-25T01:23:58.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.438+0000 7f397f7fe700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39780a2c20 0x7f39780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:58.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.438+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f39800ce140 con 0x7f3990101180 2024-04-25T01:23:58.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.438+0000 7f398dd9b700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39780a2c20 0x7f39780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:58.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.438+0000 7f398dd9b700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39780a2c20 0x7f39780a50e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f399006b880 tx=0x7f398400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:58.422 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.442+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f398001f020 con 0x7f3990101180 2024-04-25T01:23:58.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:58 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi038 2024-04-25T01:23:58.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:58 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-25T01:23:58.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:58 smithi038 ceph-mon[32841]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:23:58.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:58 smithi038 ceph-mon[32841]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:23:58.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:58 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi038 2024-04-25T01:23:58.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:58 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-25T01:23:58.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:58 smithi156 ceph-mon[32631]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:23:58.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:58 smithi156 ceph-mon[32631]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:23:58.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:58 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi038 2024-04-25T01:23:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:58 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-25T01:23:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:58 smithi016 ceph-mon[29153]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:23:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:58 smithi016 ceph-mon[29153]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:23:58.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:23:58.647+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3990062670 con 0x7f39780a2c20 2024-04-25T01:23:59.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:59 smithi038 ceph-mon[32841]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 167 KiB/s, 0 objects/s recovering 2024-04-25T01:23:59.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:59.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:59.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:23:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:59.486 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:59 smithi156 ceph-mon[32631]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 167 KiB/s, 0 objects/s recovering 2024-04-25T01:23:59.487 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:59.487 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:59.487 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:23:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:59.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:59 smithi016 ceph-mon[29153]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 167 KiB/s, 0 objects/s recovering 2024-04-25T01:23:59.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:23:59.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:23:59.535 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:23:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:59.861 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:23:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:23:59.552+0000 7f032ac39700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-25T01:24:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:00 smithi038 ceph-mon[32841]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:00.474 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:00 smithi156 ceph-mon[32631]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:00.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:00 smithi016 ceph-mon[29153]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:01 smithi038 ceph-mon[32841]: pgmap v246: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-25T01:24:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:01 smithi156 ceph-mon[32631]: pgmap v246: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-25T01:24:01.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:01 smithi016 ceph-mon[29153]: pgmap v246: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-25T01:24:02.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:02 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/3047162216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]: dispatch 2024-04-25T01:24:02.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:02 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]: dispatch 2024-04-25T01:24:02.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:02 smithi038 ceph-mon[32841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]': finished 2024-04-25T01:24:02.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:02 smithi038 ceph-mon[32841]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T01:24:02.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:02.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:02 smithi038 ceph-mon[32841]: from='client.? 172.21.15.156:0/3029392334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:24:02.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:02 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/3047162216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]: dispatch 2024-04-25T01:24:02.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:02 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]: dispatch 2024-04-25T01:24:02.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:02 smithi156 ceph-mon[32631]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]': finished 2024-04-25T01:24:02.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:02 smithi156 ceph-mon[32631]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T01:24:02.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:02.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:02 smithi156 ceph-mon[32631]: from='client.? 172.21.15.156:0/3029392334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:24:02.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:02 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/3047162216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]: dispatch 2024-04-25T01:24:02.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:02 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]: dispatch 2024-04-25T01:24:02.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:02 smithi016 ceph-mon[29153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8c5bd1d-550a-410c-8247-2dc52f893b66"}]': finished 2024-04-25T01:24:02.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:02 smithi016 ceph-mon[29153]: osdmap e77: 12 total, 11 up, 12 in 2024-04-25T01:24:02.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:02.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:02 smithi016 ceph-mon[29153]: from='client.? 172.21.15.156:0/3029392334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:24:03.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:03 smithi038 ceph-mon[32841]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-25T01:24:03.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:03 smithi156 ceph-mon[32631]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-25T01:24:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:03 smithi016 ceph-mon[29153]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-25T01:24:05.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:05 smithi038 ceph-mon[32841]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:05.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:05 smithi156 ceph-mon[32631]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:05.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:05 smithi016 ceph-mon[29153]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:07.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:07 smithi156 ceph-mon[32631]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:07.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:07 smithi038 ceph-mon[32841]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:07.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:07 smithi016 ceph-mon[29153]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:09.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:09 smithi038 ceph-mon[32841]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:09.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:24:09.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:09 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:09 smithi156 ceph-mon[32631]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:24:09.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:09 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:09.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:09 smithi016 ceph-mon[29153]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:09.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:24:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:09 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:10.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:10 smithi038 ceph-mon[32841]: Deploying daemon osd.11 on smithi156 2024-04-25T01:24:10.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:10 smithi156 ceph-mon[32631]: Deploying daemon osd.11 on smithi156 2024-04-25T01:24:10.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:10 smithi016 ceph-mon[29153]: Deploying daemon osd.11 on smithi156 2024-04-25T01:24:11.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:11 smithi038 ceph-mon[32841]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:11.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:11 smithi156 ceph-mon[32631]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:11.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:11 smithi016 ceph-mon[29153]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:13.709 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:13 smithi156 ceph-mon[32631]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:13.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:13 smithi038 ceph-mon[32841]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:13.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:13 smithi016 ceph-mon[29153]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:15.566 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:15 smithi156 ceph-mon[32631]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:15.567 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:15.567 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:15.567 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:15.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:15 smithi038 ceph-mon[32841]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:15.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:15.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:15.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:15.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:15 smithi016 ceph-mon[29153]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:17.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:17 smithi038 ceph-mon[32841]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:17.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:17 smithi156 ceph-mon[32631]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:17.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:17 smithi016 ceph-mon[29153]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:18.851 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.871+0000 7f397f7fe700 1 -- 172.21.15.156:0/2284027169 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3990062670 con 0x7f39780a2c20 2024-04-25T01:24:18.852 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 11 on host 'smithi156' 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.873+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39780a2c20 msgr2=0x7f39780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.873+0000 7f3994c09700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39780a2c20 0x7f39780a50e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f399006b880 tx=0x7f398400b3c0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 msgr2=0x7f399006b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 0x7f399006b320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f398000b500 tx=0x7f398000b810 comp rx=0 tx=0).stop 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 shutdown_connections 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f39780a2c20 0x7f39780a50e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3990102d20 0x7f399019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3990101180 0x7f399006b320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 --2- 172.21.15.156:0/2284027169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3990102380 0x7f399019d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 >> 172.21.15.156:0/2284027169 conn(0x7f39900fc720 msgr2=0x7f3990105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 shutdown_connections 2024-04-25T01:24:18.854 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:18.874+0000 7f3994c09700 1 -- 172.21.15.156:0/2284027169 wait complete. 2024-04-25T01:24:19.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:19 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:24:19.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:24:19.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:19 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:19.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:19.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:19.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:19 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:19.497 DEBUG:teuthology.orchestra.run.smithi156:osd.11> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T01:24:19.499 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T01:24:19.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:24:19.756 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:20.622 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:20 smithi156 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:24:20.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:20 smithi156 ceph-mon[32631]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T01:24:20.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:20 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:20.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:20 smithi156 ceph-mon[32631]: from='osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:24:20.623 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:20 smithi156 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:24:20.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:20 smithi016 ceph-mon[29153]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:24:20.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:20 smithi016 ceph-mon[29153]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T01:24:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:20 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:20 smithi016 ceph-mon[29153]: from='osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:24:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:20 smithi016 ceph-mon[29153]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:24:20.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:20 smithi038 ceph-mon[32841]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:24:20.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:20 smithi038 ceph-mon[32841]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T01:24:20.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:20 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:20.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:20 smithi038 ceph-mon[32841]: from='osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:24:20.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:20 smithi038 ceph-mon[32841]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T01:24:21.251 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:24:20 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:24:20.985+0000 7f3af0e07700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T01:24:21.251 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:24:21.000+0000 7f3ae8bf2700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:24:21.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:21 smithi016 ceph-mon[29153]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:21.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:21 smithi016 ceph-mon[29153]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:24:21.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:21 smithi016 ceph-mon[29153]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:24:21.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:21.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:21 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:21.628 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:21 smithi016 ceph-mon[29153]: from='osd.11 ' entity='osd.11' 2024-04-25T01:24:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 -- 172.21.15.16:0/663781596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a601022a0 msgr2=0x7f1a601026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 --2- 172.21.15.16:0/663781596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a601022a0 0x7f1a601026c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a500099c0 tx=0x7f1a50009cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 -- 172.21.15.16:0/663781596 shutdown_connections 2024-04-25T01:24:21.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 --2- 172.21.15.16:0/663781596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a60103e40 0x7f1a6010a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 --2- 172.21.15.16:0/663781596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a601022a0 0x7f1a601026c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 --2- 172.21.15.16:0/663781596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a60103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 -- 172.21.15.16:0/663781596 >> 172.21.15.16:0/663781596 conn(0x7f1a600fd840 msgr2=0x7f1a600ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 -- 172.21.15.16:0/663781596 shutdown_connections 2024-04-25T01:24:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.650+0000 7f1a6506c700 1 -- 172.21.15.16:0/663781596 wait complete. 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 Processor -- start 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 -- start start 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a601022a0 0x7f1a60198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a601989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a60103e40 0x7f1a6019f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6010dd90 con 0x7f1a60103e40 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6010dc10 con 0x7f1a601022a0 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a6506c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6010df10 con 0x7f1a601034a0 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a5f59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a60103e40 0x7f1a6019f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a5e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a601989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a5f59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a60103e40 0x7f1a6019f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44306/0 (socket says 172.21.15.16:44306) 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a5e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a601989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:59278/0 (socket says 172.21.15.16:59278) 2024-04-25T01:24:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.651+0000 7f1a5f59e700 1 -- 172.21.15.16:0/1475792149 learned_addr learned my addr 172.21.15.16:0/1475792149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5ed9d700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a601022a0 0x7f1a60198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5e59c700 1 -- 172.21.15.16:0/1475792149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a601022a0 msgr2=0x7f1a60198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5e59c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a601022a0 0x7f1a60198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5e59c700 1 -- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a60103e40 msgr2=0x7f1a6019f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5e59c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a60103e40 0x7f1a6019f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5e59c700 1 -- 172.21.15.16:0/1475792149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a601a0070 con 0x7f1a601034a0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5ed9d700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a601022a0 0x7f1a60198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a5e59c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a601989a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400ea20 tx=0x7f1a5400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400cbd0 con 0x7f1a601034a0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a601a0360 con 0x7f1a601034a0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a5400cd70 con 0x7f1a601034a0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54018680 con 0x7f1a601034a0 2024-04-25T01:24:21.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.652+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a601a0970 con 0x7f1a601034a0 2024-04-25T01:24:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.653+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a60068f20 con 0x7f1a601034a0 2024-04-25T01:24:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.654+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1a54010430 con 0x7f1a601034a0 2024-04-25T01:24:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.655+0000 7f1a47fff700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1a480a2c20 0x7f1a480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.655+0000 7f1a5ed9d700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1a480a2c20 0x7f1a480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:21.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.655+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f1a540cf110 con 0x7f1a601034a0 2024-04-25T01:24:21.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.655+0000 7f1a5ed9d700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1a480a2c20 0x7f1a480a50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000b580 tx=0x7f1a50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.658+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1a5409c590 con 0x7f1a601034a0 2024-04-25T01:24:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-mon[32631]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-mon[32631]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:24:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-mon[32631]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:24:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:21.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:21 smithi156 ceph-mon[32631]: from='osd.11 ' entity='osd.11' 2024-04-25T01:24:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.865+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f1a6002db30 con 0x7f1a601034a0 2024-04-25T01:24:21.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.866+0000 7f1a47fff700 1 -- 172.21.15.16:0/1475792149 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f1a5409be90 con 0x7f1a601034a0 2024-04-25T01:24:21.846 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1a480a2c20 msgr2=0x7f1a480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1a480a2c20 0x7f1a480a50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000b580 tx=0x7f1a50005fb0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 msgr2=0x7f1a601989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a601989a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400ea20 tx=0x7f1a5400ed30 comp rx=0 tx=0).stop 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 shutdown_connections 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1a480a2c20 0x7f1a480a50e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a60103e40 0x7f1a6019f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a601022a0 0x7f1a60198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 --2- 172.21.15.16:0/1475792149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a601034a0 0x7f1a601989a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 >> 172.21.15.16:0/1475792149 conn(0x7f1a600fd840 msgr2=0x7f1a601068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 shutdown_connections 2024-04-25T01:24:21.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:21.868+0000 7f1a6506c700 1 -- 172.21.15.16:0/1475792149 wait complete. 2024-04-25T01:24:21.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:21 smithi038 ceph-mon[32841]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:21.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:21 smithi038 ceph-mon[32841]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T01:24:21.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:21 smithi038 ceph-mon[32841]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:24:21.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:21.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:21 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:21.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:21 smithi038 ceph-mon[32841]: from='osd.11 ' entity='osd.11' 2024-04-25T01:24:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: purged_snaps scrub starts 2024-04-25T01:24:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: purged_snaps scrub ok 2024-04-25T01:24:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1475792149' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:24:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] boot 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:22.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:22 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.637 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1714008234,"num_in_osds":12,"osd_in_since":1714008241,"num_remapped_pgs":0} 2024-04-25T01:24:22.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: purged_snaps scrub starts 2024-04-25T01:24:22.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: purged_snaps scrub ok 2024-04-25T01:24:22.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1475792149' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] boot 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:22.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:22 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: purged_snaps scrub starts 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: purged_snaps scrub ok 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1475792149' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: osd.11 [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] boot 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:22.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:22.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:22.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:22.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:22 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:23.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:24:23.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:23 smithi156 ceph-mon[32631]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:23.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:23 smithi156 ceph-mon[32631]: Detected new or changed devices on smithi156 2024-04-25T01:24:23.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:23 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-25T01:24:23.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:23 smithi016 ceph-mon[29153]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:23 smithi016 ceph-mon[29153]: Detected new or changed devices on smithi156 2024-04-25T01:24:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:23 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-25T01:24:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:23.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:23 smithi038 ceph-mon[32841]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:24:23.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:23 smithi038 ceph-mon[32841]: Detected new or changed devices on smithi156 2024-04-25T01:24:23.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:23 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-25T01:24:24.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:24 smithi016 ceph-mon[29153]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T01:24:24.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:24 smithi038 ceph-mon[32841]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T01:24:24.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:24 smithi156 ceph-mon[32631]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T01:24:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 -- 172.21.15.16:0/690954159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341011d0 msgr2=0x7fb6341015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/690954159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341011d0 0x7fb6341015f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb6300099c0 tx=0x7fb630009cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 -- 172.21.15.16:0/690954159 shutdown_connections 2024-04-25T01:24:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/690954159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 0x7fb634109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/690954159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341011d0 0x7fb6341015f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/690954159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6341023d0 0x7fb634102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 -- 172.21.15.16:0/690954159 >> 172.21.15.16:0/690954159 conn(0x7fb6340fc790 msgr2=0x7fb6340febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:25.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 -- 172.21.15.16:0/690954159 shutdown_connections 2024-04-25T01:24:25.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:25 smithi016 ceph-mon[29153]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 -- 172.21.15.16:0/690954159 wait complete. 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 Processor -- start 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.761+0000 7fb63b5ec700 1 -- start start 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb63b5ec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6341011d0 0x7fb634105160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb63b5ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 0x7fb6341037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb63b5ec700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 0x7fb634103cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb63b5ec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb63410cc70 con 0x7fb6341023d0 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb63b5ec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb63410caf0 con 0x7fb6341011d0 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb63b5ec700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb63410cdf0 con 0x7fb634102d70 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 0x7fb6341037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 0x7fb6341037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44324/0 (socket says 172.21.15.16:44324) 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 -- 172.21.15.16:0/2732190395 learned_addr learned my addr 172.21.15.16:0/2732190395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb639388700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6341011d0 0x7fb634105160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb639b89700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 0x7fb634103cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 -- 172.21.15.16:0/2732190395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 msgr2=0x7fb634103cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 0x7fb634103cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 -- 172.21.15.16:0/2732190395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6341011d0 msgr2=0x7fb634105160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6341011d0 0x7fb634105160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 -- 172.21.15.16:0/2732190395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6341045a0 con 0x7fb6341023d0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb639b89700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 0x7fb634103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.762+0000 7fb638b87700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 0x7fb6341037b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c920 tx=0x7fb62800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.763+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb628007b00 con 0x7fb6341023d0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.763+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6341a25b0 con 0x7fb6341023d0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.763+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb628004ce0 con 0x7fb6341023d0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.763+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6341a2af0 con 0x7fb6341023d0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.763+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6280054b0 con 0x7fb6341023d0 2024-04-25T01:24:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.764+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb618005330 con 0x7fb6341023d0 2024-04-25T01:24:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.765+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb628004750 con 0x7fb6341023d0 2024-04-25T01:24:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.766+0000 7fb6267fc700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb6200a2c20 0x7fb6200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.766+0000 7fb639388700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb6200a2c20 0x7fb6200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.766+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb6280ceb70 con 0x7fb6341023d0 2024-04-25T01:24:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.766+0000 7fb639388700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb6200a2c20 0x7fb6200a50e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb630000c00 tx=0x7fb630005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:25.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:25 smithi156 ceph-mon[32631]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.770+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb628014030 con 0x7fb6341023d0 2024-04-25T01:24:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.977+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb6180058f0 con 0x7fb6341023d0 2024-04-25T01:24:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.977+0000 7fb6267fc700 1 -- 172.21.15.16:0/2732190395 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fb62809bed0 con 0x7fb6341023d0 2024-04-25T01:24:25.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb6200a2c20 msgr2=0x7fb6200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb6200a2c20 0x7fb6200a50e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb630000c00 tx=0x7fb630005fb0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 msgr2=0x7fb6341037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 0x7fb6341037b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c920 tx=0x7fb62800cc30 comp rx=0 tx=0).stop 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 shutdown_connections 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb6200a2c20 0x7fb6200a50e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb634102d70 0x7fb634103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6341011d0 0x7fb634105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 --2- 172.21.15.16:0/2732190395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6341023d0 0x7fb6341037b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 >> 172.21.15.16:0/2732190395 conn(0x7fb6340fc790 msgr2=0x7fb6340fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 shutdown_connections 2024-04-25T01:24:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:25.979+0000 7fb63b5ec700 1 -- 172.21.15.16:0/2732190395 wait complete. 2024-04-25T01:24:25.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:25 smithi038 ceph-mon[32841]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:26.693 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:26 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2732190395' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:24:26.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714008261,"num_in_osds":12,"osd_in_since":1714008241,"num_remapped_pgs":0} 2024-04-25T01:24:26.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:24:26.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:26 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2732190395' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:24:26.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:26 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2732190395' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:24:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:27.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:27 smithi016 ceph-mon[29153]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:27.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:27 smithi156 ceph-mon[32631]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:27.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:27 smithi038 ceph-mon[32841]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/2414665915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101130 msgr2=0x7fb1cc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/2414665915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101130 0x7fb1cc1015b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009990 tx=0x7fb1c0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/2414665915 shutdown_connections 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/2414665915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1cc101af0 0x7fb1cc10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/2414665915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1cc1071a0 0x7fb1cc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/2414665915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101130 0x7fb1cc1015b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/2414665915 >> 172.21.15.16:0/2414665915 conn(0x7fb1cc0780e0 msgr2=0x7fb1cc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/2414665915 shutdown_connections 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.860+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/2414665915 wait complete. 2024-04-25T01:24:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 Processor -- start 2024-04-25T01:24:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 -- start start 2024-04-25T01:24:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1cc101130 0x7fb1cc19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 0x7fb1cc19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1cc1071a0 0x7fb1cc198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc111340 con 0x7fb1cc101af0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc1111c0 con 0x7fb1cc101130 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1d1a0a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc1114c0 con 0x7fb1cc1071a0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1ca7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 0x7fb1cc19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1ca7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 0x7fb1cc19eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44340/0 (socket says 172.21.15.16:44340) 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1ca7fc700 1 -- 172.21.15.16:0/3071167375 learned_addr learned my addr 172.21.15.16:0/3071167375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.861+0000 7fb1caffd700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1cc101130 0x7fb1cc19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1cb7fe700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1cc1071a0 0x7fb1cc198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1ca7fc700 1 -- 172.21.15.16:0/3071167375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1cc1071a0 msgr2=0x7fb1cc198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1ca7fc700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1cc1071a0 0x7fb1cc198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1ca7fc700 1 -- 172.21.15.16:0/3071167375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1cc101130 msgr2=0x7fb1cc19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1ca7fc700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1cc101130 0x7fb1cc19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1ca7fc700 1 -- 172.21.15.16:0/3071167375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1cc198f60 con 0x7fb1cc101af0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1ca7fc700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 0x7fb1cc19eb40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0003ab0 tx=0x7fb1c0003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0027070 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1c000bdc0 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1cc1991f0 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0022ca0 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.862+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1cc1a5460 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.863+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1cc102850 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.864+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb1c002a070 con 0x7fb1cc101af0 2024-04-25T01:24:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.865+0000 7fb1d0a08700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1b40a2c20 0x7fb1b40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.865+0000 7fb1caffd700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1b40a2c20 0x7fb1b40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.865+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb1c00cfa40 con 0x7fb1cc101af0 2024-04-25T01:24:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.866+0000 7fb1caffd700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1b40a2c20 0x7fb1b40a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005950 tx=0x7fb1bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:28.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:28.870+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb1c005e740 con 0x7fb1cc101af0 2024-04-25T01:24:29.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.076+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1cc1a5630 con 0x7fb1cc101af0 2024-04-25T01:24:29.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.077+0000 7fb1d0a08700 1 -- 172.21.15.16:0/3071167375 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18433 (secure 0 0 0) 0x7fb1c0013070 con 0x7fb1cc101af0 2024-04-25T01:24:29.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:29.057 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:24:23.496690+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1b40a2c20 msgr2=0x7fb1b40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1b40a2c20 0x7fb1b40a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005950 tx=0x7fb1bc00b410 comp rx=0 tx=0).stop 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 msgr2=0x7fb1cc19eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 0x7fb1cc19eb40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0003ab0 tx=0x7fb1c0003ae0 comp rx=0 tx=0).stop 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 shutdown_connections 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1b40a2c20 0x7fb1b40a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1cc1071a0 0x7fb1cc198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1cc101130 0x7fb1cc19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 --2- 172.21.15.16:0/3071167375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1cc101af0 0x7fb1cc19eb40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 >> 172.21.15.16:0/3071167375 conn(0x7fb1cc0780e0 msgr2=0x7fb1cc100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 shutdown_connections 2024-04-25T01:24:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:29.079+0000 7fb1d1a0a700 1 -- 172.21.15.16:0/3071167375 wait complete. 2024-04-25T01:24:29.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:29 smithi016 ceph-mon[29153]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:29.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3071167375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:29.956 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:24:29.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T01:24:29.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:29 smithi038 ceph-mon[32841]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:29.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3071167375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:30.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:29 smithi156 ceph-mon[32631]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:30.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3071167375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:30.206 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:31.626 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:31 smithi016 ceph-mon[29153]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:31.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:31 smithi038 ceph-mon[32841]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:32.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:31 smithi156 ceph-mon[32631]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 -- 172.21.15.16:0/2570334797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844406ac40 msgr2=0x7f844406b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 --2- 172.21.15.16:0/2570334797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844406ac40 0x7f844406b020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84340099c0 tx=0x7f8434009cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 -- 172.21.15.16:0/2570334797 shutdown_connections 2024-04-25T01:24:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 --2- 172.21.15.16:0/2570334797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8444107590 0x7f844410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 --2- 172.21.15.16:0/2570334797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f844406ac40 0x7f844406b020 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 --2- 172.21.15.16:0/2570334797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f844406b5f0 0x7f8444106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 -- 172.21.15.16:0/2570334797 >> 172.21.15.16:0/2570334797 conn(0x7f84440781d0 msgr2=0x7f84440785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 -- 172.21.15.16:0/2570334797 shutdown_connections 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.102+0000 7f844a0d7700 1 -- 172.21.15.16:0/2570334797 wait complete. 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.103+0000 7f844a0d7700 1 Processor -- start 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.103+0000 7f844a0d7700 1 -- start start 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.103+0000 7f844a0d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f844406ac40 0x7f8444109180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.103+0000 7f844a0d7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f844406b5f0 0x7f84441096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.103+0000 7f844a0d7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 0x7f8444109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f844a0d7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8444112450 con 0x7f8444107590 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f844a0d7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84441122d0 con 0x7f844406ac40 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f844a0d7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84441125d0 con 0x7f844406b5f0 2024-04-25T01:24:32.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 0x7f8444109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 0x7f8444109c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39692/0 (socket says 172.21.15.16:39692) 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 -- 172.21.15.16:0/267644712 learned_addr learned my addr 172.21.15.16:0/267644712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f84437fe700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f844406ac40 0x7f8444109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8442ffd700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f844406b5f0 0x7f84441096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 -- 172.21.15.16:0/267644712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f844406b5f0 msgr2=0x7f84441096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f844406b5f0 0x7f84441096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 -- 172.21.15.16:0/267644712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f844406ac40 msgr2=0x7f8444109180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f844406ac40 0x7f8444109180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 -- 172.21.15.16:0/267644712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84441ab210 con 0x7f8444107590 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8443fff700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 0x7f8444109c00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ef30 tx=0x7f843c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c0092e0 con 0x7f8444107590 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84441ab4a0 con 0x7f8444107590 2024-04-25T01:24:32.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f843c004bb0 con 0x7f8444107590 2024-04-25T01:24:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.104+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c004280 con 0x7f8444107590 2024-04-25T01:24:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.105+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84441aba30 con 0x7f8444107590 2024-04-25T01:24:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.106+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8444103990 con 0x7f8444107590 2024-04-25T01:24:32.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.107+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f843c01b030 con 0x7f8444107590 2024-04-25T01:24:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.107+0000 7f8440ff9700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f842c0a2c20 0x7f842c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.108+0000 7f84437fe700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f842c0a2c20 0x7f842c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.108+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f843c0ce5e0 con 0x7f8444107590 2024-04-25T01:24:32.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.108+0000 7f84437fe700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f842c0a2c20 0x7f842c0a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8434003760 tx=0x7f8434005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:32.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.112+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f843c022020 con 0x7f8444107590 2024-04-25T01:24:32.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.315+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84441abd30 con 0x7f8444107590 2024-04-25T01:24:32.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.315+0000 7f8440ff9700 1 -- 172.21.15.16:0/267644712 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f843c010040 con 0x7f8444107590 2024-04-25T01:24:32.296 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f842c0a2c20 msgr2=0x7f842c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f842c0a2c20 0x7f842c0a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8434003760 tx=0x7f8434005fb0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 msgr2=0x7f8444109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 0x7f8444109c00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ef30 tx=0x7f843c00c3a0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 shutdown_connections 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f842c0a2c20 0x7f842c0a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8444107590 0x7f8444109c00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f844406ac40 0x7f8444109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 --2- 172.21.15.16:0/267644712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f844406b5f0 0x7f84441096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:32.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 >> 172.21.15.16:0/267644712 conn(0x7f84440781d0 msgr2=0x7f84440ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:32.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 shutdown_connections 2024-04-25T01:24:32.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:32.319+0000 7f844a0d7700 1 -- 172.21.15.16:0/267644712 wait complete. 2024-04-25T01:24:32.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:32 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/267644712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:32.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:32 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/267644712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:33.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:32 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/267644712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:33.010 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.b on smithi038 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.e on smithi038 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.h on smithi038 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.k on smithi038 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.c on smithi156 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.f on smithi156 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.i on smithi156 2024-04-25T01:24:33.011 INFO:tasks.cephadm:Adding mds.l on smithi156 2024-04-25T01:24:33.011 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l' 2024-04-25T01:24:33.264 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:24:33.637 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:33 smithi156 ceph-mon[32631]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:33.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:33 smithi016 ceph-mon[29153]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:33.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:33 smithi038 ceph-mon[32841]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:34.650 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:34 smithi156 ceph-mon[32631]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:34.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 -- 172.21.15.156:0/108947228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2220107ff0 msgr2=0x7f222010a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 --2- 172.21.15.156:0/108947228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2220107ff0 0x7f222010a3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f221800b0a0 tx=0x7f221800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 -- 172.21.15.156:0/108947228 shutdown_connections 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 --2- 172.21.15.156:0/108947228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2220107ff0 0x7f222010a3e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 --2- 172.21.15.156:0/108947228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220102d90 0x7f2220105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 --2- 172.21.15.156:0/108947228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22201056c0 0x7f2220107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 -- 172.21.15.156:0/108947228 >> 172.21.15.156:0/108947228 conn(0x7f22200fc720 msgr2=0x7f22200feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 -- 172.21.15.156:0/108947228 shutdown_connections 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 -- 172.21.15.156:0/108947228 wait complete. 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 Processor -- start 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.874+0000 7f22287d9700 1 -- start start 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2220102d90 0x7f2220198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 0x7f2220198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220107ff0 0x7f222019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f222010dd40 con 0x7f2220102d90 2024-04-25T01:24:34.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f222010dbc0 con 0x7f2220107ff0 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f222010dec0 con 0x7f22201056c0 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 0x7f2220198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 0x7f2220198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39010/0 (socket says 172.21.15.156:39010) 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 -- 172.21.15.156:0/3204746884 learned_addr learned my addr 172.21.15.156:0/3204746884 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2226d76700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220107ff0 0x7f222019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2226575700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2220102d90 0x7f2220198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 -- 172.21.15.156:0/3204746884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220107ff0 msgr2=0x7f222019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220107ff0 0x7f222019f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 -- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2220102d90 msgr2=0x7f2220198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2220102d90 0x7f2220198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 -- 172.21.15.156:0/3204746884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222019ff80 con 0x7f22201056c0 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2226575700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2220102d90 0x7f2220198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:34.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f2225d74700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 0x7f2220198860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f221000d980 tx=0x7f221000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:34.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210016070 con 0x7f22201056c0 2024-04-25T01:24:34.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f221000de70 con 0x7f22201056c0 2024-04-25T01:24:34.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.875+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22201a0270 con 0x7f22201056c0 2024-04-25T01:24:34.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.876+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000f3c0 con 0x7f22201056c0 2024-04-25T01:24:34.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.876+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22201a07b0 con 0x7f22201056c0 2024-04-25T01:24:34.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.877+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2220068f20 con 0x7f22201056c0 2024-04-25T01:24:34.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.878+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2210019070 con 0x7f22201056c0 2024-04-25T01:24:34.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.878+0000 7f22177fe700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f220c0a2c20 0x7f220c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.878+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22100ce6f0 con 0x7f22201056c0 2024-04-25T01:24:34.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.878+0000 7f2226575700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f220c0a2c20 0x7f220c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.879+0000 7f2226575700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f220c0a2c20 0x7f220c0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f221c009cb0 tx=0x7f221c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:34.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:34 smithi016 ceph-mon[29153]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:34.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:34.883+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f221009baf0 con 0x7f22201056c0 2024-04-25T01:24:34.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:34 smithi038 ceph-mon[32841]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:35.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.096+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}) -- 0x7f2220062930 con 0x7f220c0a2c20 2024-04-25T01:24:35.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.105+0000 7f22177fe700 1 -- 172.21.15.156:0/3204746884 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f2220062930 con 0x7f220c0a2c20 2024-04-25T01:24:35.086 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mds.all update... 2024-04-25T01:24:35.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.107+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f220c0a2c20 msgr2=0x7f220c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.107+0000 7f22287d9700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f220c0a2c20 0x7f220c0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f221c009cb0 tx=0x7f221c009260 comp rx=0 tx=0).stop 2024-04-25T01:24:35.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.107+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 msgr2=0x7f2220198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:35.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.107+0000 7f22287d9700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 0x7f2220198860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f221000d980 tx=0x7f221000dc90 comp rx=0 tx=0).stop 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 shutdown_connections 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f220c0a2c20 0x7f220c0a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220107ff0 0x7f222019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2220102d90 0x7f2220198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 --2- 172.21.15.156:0/3204746884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22201056c0 0x7f2220198860 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 >> 172.21.15.156:0/3204746884 conn(0x7f22200fc720 msgr2=0x7f22201036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 shutdown_connections 2024-04-25T01:24:35.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T01:24:35.108+0000 7f22287d9700 1 -- 172.21.15.156:0/3204746884 wait complete. 2024-04-25T01:24:35.725 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T01:24:35.728 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T01:24:35.771 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T01:24:35.814 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T01:24:35.858 DEBUG:teuthology.orchestra.run.smithi038:mds.b> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T01:24:35.860 DEBUG:teuthology.orchestra.run.smithi038:mds.e> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T01:24:35.903 DEBUG:teuthology.orchestra.run.smithi038:mds.h> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T01:24:35.946 DEBUG:teuthology.orchestra.run.smithi038:mds.k> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T01:24:35.988 DEBUG:teuthology.orchestra.run.smithi156:mds.c> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T01:24:35.991 DEBUG:teuthology.orchestra.run.smithi156:mds.f> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T01:24:36.033 DEBUG:teuthology.orchestra.run.smithi156:mds.i> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T01:24:36.076 DEBUG:teuthology.orchestra.run.smithi156:mds.l> sudo journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T01:24:36.120 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T01:24:36.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:24:36.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:36.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:36.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:36 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 -- 172.21.15.16:0/2405587197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101017b0 msgr2=0x7f5510101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 --2- 172.21.15.16:0/2405587197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101017b0 0x7f5510101bd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5500009900 tx=0x7f550002a410 comp rx=0 tx=0).stop 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 -- 172.21.15.16:0/2405587197 shutdown_connections 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 --2- 172.21.15.16:0/2405587197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5510103360 0x7f5510109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 --2- 172.21.15.16:0/2405587197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55101029a0 0x7f5510102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 --2- 172.21.15.16:0/2405587197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101017b0 0x7f5510101bd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 -- 172.21.15.16:0/2405587197 >> 172.21.15.16:0/2405587197 conn(0x7f55100fcd40 msgr2=0x7f55100ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 -- 172.21.15.16:0/2405587197 shutdown_connections 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.257+0000 7f5515e93640 1 -- 172.21.15.16:0/2405587197 wait complete. 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 Processor -- start 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 -- start start 2024-04-25T01:24:36.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55101017b0 0x7f5510197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 0x7f5510197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5510103360 0x7f551019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f551010d2b0 con 0x7f55101029a0 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f551010d130 con 0x7f55101017b0 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f5515e93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f551010d430 con 0x7f5510103360 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 0x7f5510197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 0x7f5510197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39704/0 (socket says 172.21.15.16:39704) 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 -- 172.21.15.16:0/2369040114 learned_addr learned my addr 172.21.15.16:0/2369040114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550ffff640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5510103360 0x7f551019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550f7fe640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55101017b0 0x7f5510197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 -- 172.21.15.16:0/2369040114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5510103360 msgr2=0x7f551019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5510103360 0x7f551019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 -- 172.21.15.16:0/2369040114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55101017b0 msgr2=0x7f5510197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55101017b0 0x7f5510197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550effd640 1 -- 172.21.15.16:0/2369040114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551019f620 con 0x7f55101029a0 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.258+0000 7f550f7fe640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55101017b0 0x7f5510197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.259+0000 7f550effd640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 0x7f5510197f40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f550400d950 tx=0x7f550400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.259+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504017070 con 0x7f55101029a0 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.259+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551019f910 con 0x7f55101029a0 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.259+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5504009c30 con 0x7f55101029a0 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.259+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550400f620 con 0x7f55101029a0 2024-04-25T01:24:36.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.259+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551019fe60 con 0x7f55101029a0 2024-04-25T01:24:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.260+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5510108ba0 con 0x7f55101029a0 2024-04-25T01:24:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.261+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f55040109a0 con 0x7f55101029a0 2024-04-25T01:24:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.262+0000 7f550cff9640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f54f80a2b90 0x7f54f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.262+0000 7f550f7fe640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f54f80a2b90 0x7f54f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.262+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55040ce7f0 con 0x7f55101029a0 2024-04-25T01:24:36.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.262+0000 7f550f7fe640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f54f80a2b90 0x7f54f80a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5500006010 tx=0x7f5500000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:36.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.266+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5504009de0 con 0x7f55101029a0 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:36.261 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:36 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:36.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:36.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:36 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:36.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.451+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f551002d320 con 0x7f55101029a0 2024-04-25T01:24:36.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.452+0000 7f550cff9640 1 -- 172.21.15.16:0/2369040114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18433 (secure 0 0 0) 0x7f550409bbf0 con 0x7f55101029a0 2024-04-25T01:24:36.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:36.433 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:24:23.496690+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:24:36.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f54f80a2b90 msgr2=0x7f54f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f54f80a2b90 0x7f54f80a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5500006010 tx=0x7f5500000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 msgr2=0x7f5510197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 0x7f5510197f40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f550400d950 tx=0x7f550400de10 comp rx=0 tx=0).stop 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 shutdown_connections 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f54f80a2b90 0x7f54f80a5050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5510103360 0x7f551019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55101029a0 0x7f5510197f40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 --2- 172.21.15.16:0/2369040114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55101017b0 0x7f5510197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 >> 172.21.15.16:0/2369040114 conn(0x7f55100fcd40 msgr2=0x7f5510105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 shutdown_connections 2024-04-25T01:24:36.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.454+0000 7f5515e93640 1 -- 172.21.15.16:0/2369040114 wait complete. 2024-04-25T01:24:36.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:24:36.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:24:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 -- 172.21.15.16:0/3369549087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 msgr2=0x7f15641098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 --2- 172.21.15.16:0/3369549087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f15641098b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f155c009ca0 tx=0x7f155c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 -- 172.21.15.16:0/3369549087 shutdown_connections 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 --2- 172.21.15.16:0/3369549087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f15641098b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 --2- 172.21.15.16:0/3369549087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1564104b30 0x7f1564106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 --2- 172.21.15.16:0/3369549087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564102200 0x7f15641045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 -- 172.21.15.16:0/3369549087 >> 172.21.15.16:0/3369549087 conn(0x7f15640fbbf0 msgr2=0x7f15640fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 -- 172.21.15.16:0/3369549087 shutdown_connections 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 -- 172.21.15.16:0/3369549087 wait complete. 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.571+0000 7f1569bdc640 1 Processor -- start 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 -- start start 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1564102200 0x7f15641979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564104b30 0x7f1564197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f156419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f156410d300 con 0x7f1564107460 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f156410d180 con 0x7f1564104b30 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1569bdc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f156410d480 con 0x7f1564102200 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f156419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f156419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39732/0 (socket says 172.21.15.16:39732) 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 -- 172.21.15.16:0/4200779815 learned_addr learned my addr 172.21.15.16:0/4200779815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f15637fe640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1564102200 0x7f15641979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1562ffd640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564104b30 0x7f1564197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 -- 172.21.15.16:0/4200779815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1564102200 msgr2=0x7f15641979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1564102200 0x7f15641979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 -- 172.21.15.16:0/4200779815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564104b30 msgr2=0x7f1564197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564104b30 0x7f1564197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1563fff640 1 -- 172.21.15.16:0/4200779815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156419f5d0 con 0x7f1564107460 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.572+0000 7f1562ffd640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564104b30 0x7f1564197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.573+0000 7f1563fff640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f156419ef60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f155c031040 tx=0x7f155c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.573+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c032ec0 con 0x7f1564107460 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.573+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f155c0044f0 con 0x7f1564107460 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.573+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156419f860 con 0x7f1564107460 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.573+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c030bb0 con 0x7f1564107460 2024-04-25T01:24:36.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.573+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156419fdb0 con 0x7f1564107460 2024-04-25T01:24:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.574+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15640fd6e0 con 0x7f1564107460 2024-04-25T01:24:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.575+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f155c004040 con 0x7f1564107460 2024-04-25T01:24:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.575+0000 7f1560ff9640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f154c0a2d30 0x7f154c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.576+0000 7f15637fe640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f154c0a2d30 0x7f154c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.576+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f155c0f0150 con 0x7f1564107460 2024-04-25T01:24:36.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.576+0000 7f15637fe640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f154c0a2d30 0x7f154c0a51f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1554009bd0 tx=0x7f1554005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:36.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.580+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f155c0bd5d0 con 0x7f1564107460 2024-04-25T01:24:36.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.763+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1564062340 con 0x7f1564107460 2024-04-25T01:24:36.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.763+0000 7f1560ff9640 1 -- 172.21.15.16:0/4200779815 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f155c0bced0 con 0x7f1564107460 2024-04-25T01:24:36.744 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f154c0a2d30 msgr2=0x7f154c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f154c0a2d30 0x7f154c0a51f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1554009bd0 tx=0x7f1554005f90 comp rx=0 tx=0).stop 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 msgr2=0x7f156419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f156419ef60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f155c031040 tx=0x7f155c003ba0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 shutdown_connections 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f154c0a2d30 0x7f154c0a51f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1564107460 0x7f156419ef60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1564104b30 0x7f1564197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 --2- 172.21.15.16:0/4200779815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1564102200 0x7f15641979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 >> 172.21.15.16:0/4200779815 conn(0x7f15640fbbf0 msgr2=0x7f1564102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 shutdown_connections 2024-04-25T01:24:36.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.765+0000 7f1569bdc640 1 -- 172.21.15.16:0/4200779815 wait complete. 2024-04-25T01:24:36.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 -- 172.21.15.16:0/2236008393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c102250 msgr2=0x7f941c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 --2- 172.21.15.16:0/2236008393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c102250 0x7f941c108ae0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9414009ca0 tx=0x7f941402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 -- 172.21.15.16:0/2236008393 shutdown_connections 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 --2- 172.21.15.16:0/2236008393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c102250 0x7f941c108ae0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 --2- 172.21.15.16:0/2236008393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f941c101890 0x7f941c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 --2- 172.21.15.16:0/2236008393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f941c1006a0 0x7f941c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 -- 172.21.15.16:0/2236008393 >> 172.21.15.16:0/2236008393 conn(0x7f941c0fbc30 msgr2=0x7f941c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.877+0000 7f9422676640 1 -- 172.21.15.16:0/2236008393 shutdown_connections 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 -- 172.21.15.16:0/2236008393 wait complete. 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 Processor -- start 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 -- start start 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 0x7f941c1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f941c101890 0x7f941c197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f941c102250 0x7f941c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f941c10c1b0 con 0x7f941c1006a0 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f941c10c030 con 0x7f941c101890 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.878+0000 7f9422676640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f941c10c330 con 0x7f941c102250 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 0x7f941c1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 0x7f941c1978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39748/0 (socket says 172.21.15.16:39748) 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 -- 172.21.15.16:0/2006450293 learned_addr learned my addr 172.21.15.16:0/2006450293 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f9420c13640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f941c102250 0x7f941c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941b7fe640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f941c101890 0x7f941c197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 -- 172.21.15.16:0/2006450293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f941c102250 msgr2=0x7f941c19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f941c102250 0x7f941c19ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 -- 172.21.15.16:0/2006450293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f941c101890 msgr2=0x7f941c197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f941c101890 0x7f941c197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 -- 172.21.15.16:0/2006450293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941c19f4d0 con 0x7f941c1006a0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f941bfff640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 0x7f941c1978b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f940c00c970 tx=0x7f940c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c007c10 con 0x7f941c1006a0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f940c007db0 con 0x7f941c1006a0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c00f870 con 0x7f941c1006a0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941c19f7c0 con 0x7f941c1006a0 2024-04-25T01:24:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.879+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f941c19fd60 con 0x7f941c1006a0 2024-04-25T01:24:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.880+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f941c19ffc0 con 0x7f941c1006a0 2024-04-25T01:24:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.881+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f940c00fa10 con 0x7f941c1006a0 2024-04-25T01:24:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.882+0000 7f94197fa640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f94040a2c10 0x7f94040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:36.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.882+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f940c0cef70 con 0x7f941c1006a0 2024-04-25T01:24:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.882+0000 7f941b7fe640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f94040a2c10 0x7f94040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:36.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.882+0000 7f941b7fe640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f94040a2c10 0x7f94040a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f941c198dc0 tx=0x7f9410005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:36.886+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f940c09c400 con 0x7f941c1006a0 2024-04-25T01:24:37.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.111+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f941c1a0230 con 0x7f941c1006a0 2024-04-25T01:24:37.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.111+0000 7f94197fa640 1 -- 172.21.15.16:0/2006450293 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f940c09bd00 con 0x7f941c1006a0 2024-04-25T01:24:37.091 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:37.092 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T01:24:37.092 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-04-25T01:24:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f94040a2c10 msgr2=0x7f94040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f94040a2c10 0x7f94040a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f941c198dc0 tx=0x7f9410005f90 comp rx=0 tx=0).stop 2024-04-25T01:24:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 msgr2=0x7f941c1978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 0x7f941c1978b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f940c00c970 tx=0x7f940c00ce30 comp rx=0 tx=0).stop 2024-04-25T01:24:37.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 shutdown_connections 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f94040a2c10 0x7f94040a50d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f941c102250 0x7f941c19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f941c101890 0x7f941c197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 --2- 172.21.15.16:0/2006450293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f941c1006a0 0x7f941c1978b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 >> 172.21.15.16:0/2006450293 conn(0x7f941c0fbc30 msgr2=0x7f941c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 shutdown_connections 2024-04-25T01:24:37.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:37.113+0000 7f9422676640 1 -- 172.21.15.16:0/2006450293 wait complete. 2024-04-25T01:24:37.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:37 smithi156 ceph-mon[32631]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:37.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:37 smithi156 ceph-mon[32631]: Deploying daemon mds.l on smithi156 2024-04-25T01:24:37.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:37 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2369040114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:37.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:37 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/4200779815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:37.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:37 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2006450293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:37.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:37 smithi016 ceph-mon[29153]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:37.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:37 smithi016 ceph-mon[29153]: Deploying daemon mds.l on smithi156 2024-04-25T01:24:37.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:37 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2369040114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:37.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:37 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4200779815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:37.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:37 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2006450293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:37.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:37 smithi038 ceph-mon[32841]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:37.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:37 smithi038 ceph-mon[32841]: Deploying daemon mds.l on smithi156 2024-04-25T01:24:37.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:37 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2369040114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:24:37.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:37 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/4200779815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:24:37.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:37 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2006450293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:38.199 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 systemd[1]: Starting Ceph mds.l for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:38.469 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 podman[62997]: 2024-04-25 01:24:38.121994923 +0000 UTC m=+0.018355228 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:38.469 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 podman[62997]: 2024-04-25 01:24:38.237710428 +0000 UTC m=+0.134070748 container create 4cf67c1f405b13ebd9e155b5ac6ba4d9a890424d08b9f9c1f7f67ad480343bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T01:24:38.469 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 podman[62997]: 2024-04-25 01:24:38.49020529 +0000 UTC m=+0.386565568 container init 4cf67c1f405b13ebd9e155b5ac6ba4d9a890424d08b9f9c1f7f67ad480343bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T01:24:38.751 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 podman[62997]: 2024-04-25 01:24:38.49283822 +0000 UTC m=+0.389198509 container start 4cf67c1f405b13ebd9e155b5ac6ba4d9a890424d08b9f9c1f7f67ad480343bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:24:38.751 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 ceph-mds[63016]: -- 172.21.15.156:0/2978005721 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b18e3e5a40 con 0x55b18e3b8800 2024-04-25T01:24:38.751 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l[63012]: starting mds.l at 2024-04-25T01:24:38.751 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 bash[62997]: 4cf67c1f405b13ebd9e155b5ac6ba4d9a890424d08b9f9c1f7f67ad480343bcb 2024-04-25T01:24:38.751 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:24:38 smithi156 systemd[1]: Started Ceph mds.l for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:39.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:39.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 -- 172.21.15.16:0/3942864002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 msgr2=0x7f2958108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:39.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 --2- 172.21.15.16:0/3942864002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f2958108ae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2950009ca0 tx=0x7f295002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 -- 172.21.15.16:0/3942864002 shutdown_connections 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 --2- 172.21.15.16:0/3942864002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f2958108ae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 --2- 172.21.15.16:0/3942864002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2958101890 0x7f2958101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 --2- 172.21.15.16:0/3942864002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29581006a0 0x7f2958100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 -- 172.21.15.16:0/3942864002 >> 172.21.15.16:0/3942864002 conn(0x7f29580fbc30 msgr2=0x7f29580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 -- 172.21.15.16:0/3942864002 shutdown_connections 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 -- 172.21.15.16:0/3942864002 wait complete. 2024-04-25T01:24:39.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.240+0000 7f295d390640 1 Processor -- start 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 -- start start 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29581006a0 0x7f2958197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2958101890 0x7f2958197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f295819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f295810d2f0 con 0x7f2958102250 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f295810d170 con 0x7f2958101890 2024-04-25T01:24:39.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f295d390640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f295810d470 con 0x7f29581006a0 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f295819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f295819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39760/0 (socket says 172.21.15.16:39760) 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 -- 172.21.15.16:0/624380005 learned_addr learned my addr 172.21.15.16:0/624380005 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f2956ffd640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29581006a0 0x7f2958197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29567fc640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2958101890 0x7f2958197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 -- 172.21.15.16:0/624380005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29581006a0 msgr2=0x7f2958197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29581006a0 0x7f2958197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 -- 172.21.15.16:0/624380005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2958101890 msgr2=0x7f2958197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2958101890 0x7f2958197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.241+0000 7f29577fe640 1 -- 172.21.15.16:0/624380005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295819f580 con 0x7f2958102250 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.242+0000 7f29577fe640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f295819ef10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2950009dc0 tx=0x7f2950003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.242+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950032ec0 con 0x7f2958102250 2024-04-25T01:24:39.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.242+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295819f810 con 0x7f2958102250 2024-04-25T01:24:39.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.242+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29500044f0 con 0x7f2958102250 2024-04-25T01:24:39.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.242+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950030ba0 con 0x7f2958102250 2024-04-25T01:24:39.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.242+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f295819fdb0 con 0x7f2958102250 2024-04-25T01:24:39.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.243+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2958107960 con 0x7f2958102250 2024-04-25T01:24:39.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.244+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f29500046a0 con 0x7f2958102250 2024-04-25T01:24:39.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.245+0000 7f293ffff640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29400a2d30 0x7f29400a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:39.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.245+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29500f0250 con 0x7f2958102250 2024-04-25T01:24:39.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.245+0000 7f2956ffd640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29400a2d30 0x7f29400a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:39.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.245+0000 7f2956ffd640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29400a2d30 0x7f29400a51f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29581016f0 tx=0x7f294800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:39.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.249+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f29500bd650 con 0x7f2958102250 2024-04-25T01:24:39.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.477+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2958062340 con 0x7f2958102250 2024-04-25T01:24:39.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.478+0000 7f293ffff640 1 -- 172.21.15.16:0/624380005 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f29500bcf50 con 0x7f2958102250 2024-04-25T01:24:39.458 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:39.458 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:24:39.459 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-04-25T01:24:39.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29400a2d30 msgr2=0x7f29400a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:39.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29400a2d30 0x7f29400a51f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29581016f0 tx=0x7f294800b390 comp rx=0 tx=0).stop 2024-04-25T01:24:39.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 msgr2=0x7f295819ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:39.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f295819ef10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2950009dc0 tx=0x7f2950003ba0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 shutdown_connections 2024-04-25T01:24:39.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29400a2d30 0x7f29400a51f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2958102250 0x7f295819ef10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2958101890 0x7f2958197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 --2- 172.21.15.16:0/624380005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29581006a0 0x7f2958197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 >> 172.21.15.16:0/624380005 conn(0x7f29580fbc30 msgr2=0x7f2958104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 shutdown_connections 2024-04-25T01:24:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:39.480+0000 7f295d390640 1 -- 172.21.15.16:0/624380005 wait complete. 2024-04-25T01:24:39.620 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:39.620 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6832/1629519202,v1:172.21.15.156:6833/1629519202] up:boot 2024-04-25T01:24:39.620 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: fsmap 1 up:standby 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:39.621 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:39 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/624380005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:39.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:39.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6832/1629519202,v1:172.21.15.156:6833/1629519202] up:boot 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: fsmap 1 up:standby 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:39 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/624380005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:40.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:40.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6832/1629519202,v1:172.21.15.156:6833/1629519202] up:boot 2024-04-25T01:24:40.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: fsmap 1 up:standby 2024-04-25T01:24:40.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:40.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:39 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/624380005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:40.702 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:40 smithi038 ceph-mon[32841]: Deploying daemon mds.k on smithi038 2024-04-25T01:24:40.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:40 smithi016 ceph-mon[29153]: Deploying daemon mds.k on smithi038 2024-04-25T01:24:40.966 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:40 smithi038 systemd[1]: Starting Ceph mds.k for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:41.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:40 smithi156 ceph-mon[32631]: Deploying daemon mds.k on smithi038 2024-04-25T01:24:41.377 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 podman[61909]: 2024-04-25 01:24:41.034692129 +0000 UTC m=+0.018973485 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:41.377 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 podman[61909]: 2024-04-25 01:24:41.266305103 +0000 UTC m=+0.250586486 container create b423c9ba0b88e50b7b10fcbec36bc9beb5e5ea7f8eb3795b5e3e855baf846b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:24:41.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:41.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2505783367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58101890 msgr2=0x7f2b58101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2505783367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58101890 0x7f2b58101d10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009900 tx=0x7f2b4c02a410 comp rx=0 tx=0).stop 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2505783367 shutdown_connections 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2505783367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b58102250 0x7f2b58108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2505783367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b58101890 0x7f2b58101d10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2505783367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b581006a0 0x7f2b58100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2505783367 >> 172.21.15.16:0/2505783367 conn(0x7f2b580fbc30 msgr2=0x7f2b580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2505783367 shutdown_connections 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.606+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2505783367 wait complete. 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 Processor -- start 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 -- start start 2024-04-25T01:24:41.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 0x7f2b581979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b58101890 0x7f2b58197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b58102250 0x7f2b5819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5810c220 con 0x7f2b581006a0 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5810c0a0 con 0x7f2b58102250 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5e68c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5810c3a0 con 0x7f2b58101890 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 0x7f2b581979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 0x7f2b581979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39770/0 (socket says 172.21.15.16:39770) 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 -- 172.21.15.16:0/2279192370 learned_addr learned my addr 172.21.15.16:0/2279192370 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b577fe640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b58101890 0x7f2b58197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b5cc29640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b58102250 0x7f2b5819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 -- 172.21.15.16:0/2279192370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b58101890 msgr2=0x7f2b58197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b58101890 0x7f2b58197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 -- 172.21.15.16:0/2279192370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b58102250 msgr2=0x7f2b5819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b58102250 0x7f2b5819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.607+0000 7f2b57fff640 1 -- 172.21.15.16:0/2279192370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5819f610 con 0x7f2b581006a0 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.608+0000 7f2b57fff640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 0x7f2b581979f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800d950 tx=0x7f2b4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.608+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b48017070 con 0x7f2b581006a0 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.608+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b48009c30 con 0x7f2b581006a0 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.608+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5819f900 con 0x7f2b581006a0 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.608+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4800f620 con 0x7f2b581006a0 2024-04-25T01:24:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.608+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b5819fdd0 con 0x7f2b581006a0 2024-04-25T01:24:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.609+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b5819ffd0 con 0x7f2b581006a0 2024-04-25T01:24:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.611+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2b4801a070 con 0x7f2b581006a0 2024-04-25T01:24:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.611+0000 7f2b557fa640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2b400a2b90 0x7f2b400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.611+0000 7f2b577fe640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2b400a2b90 0x7f2b400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.611+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2b480ce580 con 0x7f2b581006a0 2024-04-25T01:24:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.612+0000 7f2b577fe640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2b400a2b90 0x7f2b400a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0314e0 tx=0x7f2b4c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.615+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b48009de0 con 0x7f2b581006a0 2024-04-25T01:24:41.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:41 smithi038 ceph-mon[32841]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:41.717 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 podman[61909]: 2024-04-25 01:24:41.502454357 +0000 UTC m=+0.486735701 container init b423c9ba0b88e50b7b10fcbec36bc9beb5e5ea7f8eb3795b5e3e855baf846b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:24:41.717 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 podman[61909]: 2024-04-25 01:24:41.505567024 +0000 UTC m=+0.489848368 container start b423c9ba0b88e50b7b10fcbec36bc9beb5e5ea7f8eb3795b5e3e855baf846b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T01:24:41.717 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k[61924]: starting mds.k at 2024-04-25T01:24:41.717 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 bash[61909]: b423c9ba0b88e50b7b10fcbec36bc9beb5e5ea7f8eb3795b5e3e855baf846b5f 2024-04-25T01:24:41.717 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:24:41 smithi038 systemd[1]: Started Ceph mds.k for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:41.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.854+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b581a01a0 con 0x7f2b581006a0 2024-04-25T01:24:41.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.854+0000 7f2b557fa640 1 -- 172.21.15.16:0/2279192370 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f2b4809b330 con 0x7f2b581006a0 2024-04-25T01:24:41.835 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:41.835 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:24:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.856+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2b400a2b90 msgr2=0x7f2b400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.856+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2b400a2b90 0x7f2b400a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0314e0 tx=0x7f2b4c005a20 comp rx=0 tx=0).stop 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.856+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 msgr2=0x7f2b581979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.856+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 0x7f2b581979f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800d950 tx=0x7f2b4800de10 comp rx=0 tx=0).stop 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.856+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 shutdown_connections 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2b400a2b90 0x7f2b400a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b58102250 0x7f2b5819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b58101890 0x7f2b58197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 --2- 172.21.15.16:0/2279192370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b581006a0 0x7f2b581979f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 >> 172.21.15.16:0/2279192370 conn(0x7f2b580fbc30 msgr2=0x7f2b58069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 shutdown_connections 2024-04-25T01:24:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:41.857+0000 7f2b5e68c640 1 -- 172.21.15.16:0/2279192370 wait complete. 2024-04-25T01:24:41.847 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:41 smithi016 ceph-mon[29153]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:41.902 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:41 smithi156 ceph-mon[32631]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: Deploying daemon mds.j on smithi016 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2279192370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6832/3531176332,v1:172.21.15.38:6833/3531176332] up:boot 2024-04-25T01:24:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: fsmap 2 up:standby 2024-04-25T01:24:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: Deploying daemon mds.j on smithi016 2024-04-25T01:24:42.922 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2279192370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:42.923 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6832/3531176332,v1:172.21.15.38:6833/3531176332] up:boot 2024-04-25T01:24:42.923 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: fsmap 2 up:standby 2024-04-25T01:24:42.923 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:24:43.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:43.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:43.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: Deploying daemon mds.j on smithi016 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2279192370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6832/3531176332,v1:172.21.15.38:6833/3531176332] up:boot 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: fsmap 2 up:standby 2024-04-25T01:24:43.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:24:43.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 -- 172.21.15.16:0/4171612536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90102230 msgr2=0x7faf90108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 --2- 172.21.15.16:0/4171612536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90102230 0x7faf90108ac0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faf88009ca0 tx=0x7faf8802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 -- 172.21.15.16:0/4171612536 shutdown_connections 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 --2- 172.21.15.16:0/4171612536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90102230 0x7faf90108ac0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 --2- 172.21.15.16:0/4171612536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf90101870 0x7faf90101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 --2- 172.21.15.16:0/4171612536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90100680 0x7faf90100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.058+0000 7faf97e9e640 1 -- 172.21.15.16:0/4171612536 >> 172.21.15.16:0/4171612536 conn(0x7faf900fbbf0 msgr2=0x7faf900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 -- 172.21.15.16:0/4171612536 shutdown_connections 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 -- 172.21.15.16:0/4171612536 wait complete. 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 Processor -- start 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 -- start start 2024-04-25T01:24:44.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 0x7faf90069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf90101870 0x7faf9019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90102230 0x7faf9019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c1e0 con 0x7faf90100680 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c060 con 0x7faf90102230 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf97e9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c360 con 0x7faf90101870 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf95c3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 0x7faf90069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf95c3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 0x7faf90069460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39822/0 (socket says 172.21.15.16:39822) 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.059+0000 7faf95c3a640 1 -- 172.21.15.16:0/531361532 learned_addr learned my addr 172.21.15.16:0/531361532 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90102230 0x7faf9019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf95439640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf90101870 0x7faf9019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 -- 172.21.15.16:0/531361532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf90101870 msgr2=0x7faf9019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf90101870 0x7faf9019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 -- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 msgr2=0x7faf90069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 0x7faf90069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 -- 172.21.15.16:0/531361532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9019f580 con 0x7faf90102230 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf95c3a640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 0x7faf90069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf9643b640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90102230 0x7faf9019ecc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf8802ade0 tx=0x7faf88032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88003da0 con 0x7faf90102230 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf88040cd0 con 0x7faf90102230 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.060+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf90196450 con 0x7faf90102230 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.061+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf880305c0 con 0x7faf90102230 2024-04-25T01:24:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.061+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf901969c0 con 0x7faf90102230 2024-04-25T01:24:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.062+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf901079a0 con 0x7faf90102230 2024-04-25T01:24:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.063+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7faf88004160 con 0x7faf90102230 2024-04-25T01:24:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.063+0000 7faf86ffd640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf7c0a2c10 0x7faf7c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.063+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faf880f4320 con 0x7faf90102230 2024-04-25T01:24:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.063+0000 7faf95c3a640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf7c0a2c10 0x7faf7c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.064+0000 7faf95c3a640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf7c0a2c10 0x7faf7c0a50d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0104e0 tx=0x7faf8c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.068+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf880c1720 con 0x7faf90102230 2024-04-25T01:24:44.110 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:43 smithi016 systemd[1]: Starting Ceph mds.j for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:44.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.297+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf9002d380 con 0x7faf90102230 2024-04-25T01:24:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.298+0000 7faf86ffd640 1 -- 172.21.15.16:0/531361532 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7faf880c1020 con 0x7faf90102230 2024-04-25T01:24:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:44.278 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:24:44.278 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-04-25T01:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf7c0a2c10 msgr2=0x7faf7c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf7c0a2c10 0x7faf7c0a50d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0104e0 tx=0x7faf8c00b410 comp rx=0 tx=0).stop 2024-04-25T01:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90102230 msgr2=0x7faf9019ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90102230 0x7faf9019ecc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf8802ade0 tx=0x7faf88032ec0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 shutdown_connections 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf7c0a2c10 0x7faf7c0a50d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf90102230 0x7faf9019ecc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf90101870 0x7faf9019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 --2- 172.21.15.16:0/531361532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf90100680 0x7faf90069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 >> 172.21.15.16:0/531361532 conn(0x7faf900fbbf0 msgr2=0x7faf90104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 shutdown_connections 2024-04-25T01:24:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:44.300+0000 7faf97e9e640 1 -- 172.21.15.16:0/531361532 wait complete. 2024-04-25T01:24:44.473 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 podman[69264]: 2024-04-25 01:24:44.119842862 +0000 UTC m=+0.018961785 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:44.473 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 podman[69264]: 2024-04-25 01:24:44.240782687 +0000 UTC m=+0.139901583 container create 8d26326a4cc66fc870eef59cbb7c2f34ea1b1713c06885d3ee04ff2fab2006b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T01:24:44.473 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 podman[69264]: 2024-04-25 01:24:44.493878516 +0000 UTC m=+0.392997413 container init 8d26326a4cc66fc870eef59cbb7c2f34ea1b1713c06885d3ee04ff2fab2006b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T01:24:44.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:44 smithi016 ceph-mon[29153]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:44.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:44 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/531361532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:44.861 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 podman[69264]: 2024-04-25 01:24:44.496754127 +0000 UTC m=+0.395873024 container start 8d26326a4cc66fc870eef59cbb7c2f34ea1b1713c06885d3ee04ff2fab2006b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5) 2024-04-25T01:24:44.861 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 ceph-mds[69285]: -- 172.21.15.16:0/2590186052 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56451b04fa40 con 0x56451b022c00 2024-04-25T01:24:44.861 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j[69281]: starting mds.j at 2024-04-25T01:24:44.861 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 bash[69264]: 8d26326a4cc66fc870eef59cbb7c2f34ea1b1713c06885d3ee04ff2fab2006b7 2024-04-25T01:24:44.861 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:24:44 smithi016 systemd[1]: Started Ceph mds.j for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:44.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:44 smithi038 ceph-mon[32841]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:44.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:44 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/531361532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:45.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:44 smithi156 ceph-mon[32631]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:45.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:44 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/531361532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:45.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.16:6834/690842306,v1:172.21.15.16:6835/690842306] up:boot 2024-04-25T01:24:45.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: fsmap 3 up:standby 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:45.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:45 smithi038 ceph-mon[32841]: Deploying daemon mds.d on smithi016 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.16:6834/690842306,v1:172.21.15.16:6835/690842306] up:boot 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: fsmap 3 up:standby 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:46.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:45 smithi156 ceph-mon[32631]: Deploying daemon mds.d on smithi016 2024-04-25T01:24:46.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.16:6834/690842306,v1:172.21.15.16:6835/690842306] up:boot 2024-04-25T01:24:46.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: fsmap 3 up:standby 2024-04-25T01:24:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:24:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:46.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:46.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:46.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:46.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:45 smithi016 ceph-mon[29153]: Deploying daemon mds.d on smithi016 2024-04-25T01:24:46.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:46.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 -- 172.21.15.16:0/1548177338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 msgr2=0x7f0170108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 --2- 172.21.15.16:0/1548177338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f0170108ad0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0168009ca0 tx=0x7f016802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 -- 172.21.15.16:0/1548177338 shutdown_connections 2024-04-25T01:24:46.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 --2- 172.21.15.16:0/1548177338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f0170108ad0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 --2- 172.21.15.16:0/1548177338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 0x7f0170101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 --2- 172.21.15.16:0/1548177338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0170100690 0x7f0170100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 -- 172.21.15.16:0/1548177338 >> 172.21.15.16:0/1548177338 conn(0x7f01700fbc20 msgr2=0x7f01700fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 -- 172.21.15.16:0/1548177338 shutdown_connections 2024-04-25T01:24:46.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.431+0000 7f0177f5a640 1 -- 172.21.15.16:0/1548177338 wait complete. 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 Processor -- start 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 -- start start 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0170100690 0x7f017006a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 0x7f017019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f017019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f017010c1c0 con 0x7f0170102240 2024-04-25T01:24:46.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f017010c040 con 0x7f0170101880 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0177f5a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f017010c340 con 0x7f0170100690 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f017019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f017019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39854/0 (socket says 172.21.15.16:39854) 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 -- 172.21.15.16:0/856583112 learned_addr learned my addr 172.21.15.16:0/856583112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f0175cf6640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0170100690 0x7f017006a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01754f5640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 0x7f017019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 -- 172.21.15.16:0/856583112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0170100690 msgr2=0x7f017006a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0170100690 0x7f017006a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 -- 172.21.15.16:0/856583112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 msgr2=0x7f017019c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 0x7f017019c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 -- 172.21.15.16:0/856583112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017019f600 con 0x7f0170102240 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01754f5640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 0x7f017019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.432+0000 7f01764f7640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f017019ecf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0168009dc0 tx=0x7f0168003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.433+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168032ec0 con 0x7f0170102240 2024-04-25T01:24:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.433+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017019f830 con 0x7f0170102240 2024-04-25T01:24:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.433+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01680044f0 con 0x7f0170102240 2024-04-25T01:24:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.433+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168030d40 con 0x7f0170102240 2024-04-25T01:24:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.433+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017019fd80 con 0x7f0170102240 2024-04-25T01:24:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.434+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0170107a80 con 0x7f0170102240 2024-04-25T01:24:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.435+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0168004040 con 0x7f0170102240 2024-04-25T01:24:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.435+0000 7f0166ffd640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f015c0a2d30 0x7f015c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.436+0000 7f0175cf6640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f015c0a2d30 0x7f015c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.436+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f01680f0260 con 0x7f0170102240 2024-04-25T01:24:46.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.436+0000 7f0175cf6640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f015c0a2d30 0x7f015c0a51f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f016c005fd0 tx=0x7f016c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.440+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f01680bd660 con 0x7f0170102240 2024-04-25T01:24:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.674+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f017002d390 con 0x7f0170102240 2024-04-25T01:24:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.675+0000 7f0166ffd640 1 -- 172.21.15.16:0/856583112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f01680bcf60 con 0x7f0170102240 2024-04-25T01:24:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:46.655 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:24:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-04-25T01:24:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f015c0a2d30 msgr2=0x7f015c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f015c0a2d30 0x7f015c0a51f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f016c005fd0 tx=0x7f016c005850 comp rx=0 tx=0).stop 2024-04-25T01:24:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 msgr2=0x7f017019ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f017019ecf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0168009dc0 tx=0x7f0168003ba0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 shutdown_connections 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f015c0a2d30 0x7f015c0a51f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0170102240 0x7f017019ecf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0170101880 0x7f017019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 --2- 172.21.15.16:0/856583112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0170100690 0x7f017006a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 >> 172.21.15.16:0/856583112 conn(0x7f01700fbc20 msgr2=0x7f0170104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 shutdown_connections 2024-04-25T01:24:46.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:46.677+0000 7f0177f5a640 1 -- 172.21.15.16:0/856583112 wait complete. 2024-04-25T01:24:46.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:46 smithi016 ceph-mon[29153]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:46.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:46 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/856583112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:46.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:46 smithi038 ceph-mon[32841]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:46.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:46 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/856583112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:47.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:46 smithi156 ceph-mon[32631]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:47.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:46 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/856583112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:47.860 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:24:47 smithi016 systemd[1]: Started Ceph mds.d for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:48.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 -- 172.21.15.16:0/1319577976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 msgr2=0x7f1424101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 --2- 172.21.15.16:0/1319577976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424101b30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1414009900 tx=0x7f141402a410 comp rx=0 tx=0).stop 2024-04-25T01:24:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 -- 172.21.15.16:0/1319577976 shutdown_connections 2024-04-25T01:24:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 --2- 172.21.15.16:0/1319577976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 0x7f1424108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 --2- 172.21.15.16:0/1319577976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424101b30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 --2- 172.21.15.16:0/1319577976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14241004c0 0x7f14241008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 -- 172.21.15.16:0/1319577976 >> 172.21.15.16:0/1319577976 conn(0x7f14240fba50 msgr2=0x7f14240fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 -- 172.21.15.16:0/1319577976 shutdown_connections 2024-04-25T01:24:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.808+0000 7f142ad9e640 1 -- 172.21.15.16:0/1319577976 wait complete. 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 Processor -- start 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 -- start start 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14241004c0 0x7f14241977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 0x7f142419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f142410c050 con 0x7f14241016b0 2024-04-25T01:24:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f142410bed0 con 0x7f14241004c0 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142ad9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f142410c1d0 con 0x7f1424102070 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39898/0 (socket says 172.21.15.16:39898) 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 -- 172.21.15.16:0/3212183213 learned_addr learned my addr 172.21.15.16:0/3212183213 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142a59d640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 0x7f142419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f1429d9c640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14241004c0 0x7f14241977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 -- 172.21.15.16:0/3212183213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 msgr2=0x7f142419ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 0x7f142419ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 -- 172.21.15.16:0/3212183213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14241004c0 msgr2=0x7f14241977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14241004c0 0x7f14241977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 -- 172.21.15.16:0/3212183213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142419f400 con 0x7f14241016b0 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142a59d640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 0x7f142419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.809+0000 7f142959b640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424197d20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f14140314e0 tx=0x7f1414003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.810+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141403f070 con 0x7f14241016b0 2024-04-25T01:24:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.810+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1414031c40 con 0x7f14241016b0 2024-04-25T01:24:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.810+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142419f690 con 0x7f14241016b0 2024-04-25T01:24:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.810+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414035030 con 0x7f14241016b0 2024-04-25T01:24:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.810+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142419fb80 con 0x7f14241016b0 2024-04-25T01:24:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.811+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f142419fea0 con 0x7f14241016b0 2024-04-25T01:24:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.812+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1414042070 con 0x7f14241016b0 2024-04-25T01:24:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.812+0000 7f141affd640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f14100a2c10 0x7f14100a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.813+0000 7f1429d9c640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f14100a2c10 0x7f14100a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.813+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f14140efb60 con 0x7f14241016b0 2024-04-25T01:24:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.813+0000 7f1429d9c640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f14100a2c10 0x7f14100a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1420005d00 tx=0x7f1420005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:48.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:48.816+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f14140bcec0 con 0x7f14241016b0 2024-04-25T01:24:48.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: Deploying daemon mds.e on smithi038 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.16:6836/2204481226,v1:172.21.15.16:6837/2204481226] up:boot 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: fsmap 4 up:standby 2024-04-25T01:24:48.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: Deploying daemon mds.e on smithi038 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.16:6836/2204481226,v1:172.21.15.16:6837/2204481226] up:boot 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: fsmap 4 up:standby 2024-04-25T01:24:49.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:24:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.040+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14241a0070 con 0x7f14241016b0 2024-04-25T01:24:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.040+0000 7f141affd640 1 -- 172.21.15.16:0/3212183213 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f14140bc7c0 con 0x7f14241016b0 2024-04-25T01:24:49.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:49.021 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:24:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-04-25T01:24:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f14100a2c10 msgr2=0x7f14100a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f14100a2c10 0x7f14100a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1420005d00 tx=0x7f1420005c50 comp rx=0 tx=0).stop 2024-04-25T01:24:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 msgr2=0x7f1424197d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424197d20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f14140314e0 tx=0x7f1414003c10 comp rx=0 tx=0).stop 2024-04-25T01:24:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 shutdown_connections 2024-04-25T01:24:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f14100a2c10 0x7f14100a50d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1424102070 0x7f142419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14241016b0 0x7f1424197d20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 --2- 172.21.15.16:0/3212183213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14241004c0 0x7f14241977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 >> 172.21.15.16:0/3212183213 conn(0x7f14240fba50 msgr2=0x7f1424069ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 shutdown_connections 2024-04-25T01:24:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:49.043+0000 7f142ad9e640 1 -- 172.21.15.16:0/3212183213 wait complete. 2024-04-25T01:24:49.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:49.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:49.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:49.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:49.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:49.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:49.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:49.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: Deploying daemon mds.e on smithi038 2024-04-25T01:24:49.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.16:6836/2204481226,v1:172.21.15.16:6837/2204481226] up:boot 2024-04-25T01:24:49.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: fsmap 4 up:standby 2024-04-25T01:24:49.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:24:49.709 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:49 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3212183213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:49.966 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:49 smithi038 systemd[1]: Starting Ceph mds.e for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:50.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:49 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3212183213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:49 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3212183213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:50.398 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 podman[62227]: 2024-04-25 01:24:50.048353048 +0000 UTC m=+0.018826290 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:50.398 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 podman[62227]: 2024-04-25 01:24:50.190361048 +0000 UTC m=+0.160834276 container create 6ffe41c2eb68c3f63b990b221c57b59916bbd003808cb8b009bdc666ebaa0a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T01:24:50.398 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 podman[62227]: 2024-04-25 01:24:50.41833029 +0000 UTC m=+0.388803518 container init 6ffe41c2eb68c3f63b990b221c57b59916bbd003808cb8b009bdc666ebaa0a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-25T01:24:50.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:50.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:50.718 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 podman[62227]: 2024-04-25 01:24:50.421483047 +0000 UTC m=+0.391956280 container start 6ffe41c2eb68c3f63b990b221c57b59916bbd003808cb8b009bdc666ebaa0a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:24:50.718 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-mds[62246]: -- 172.21.15.38:0/2968976885 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e563603a40 con 0x55e5635d6c00 2024-04-25T01:24:50.718 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e[62242]: starting mds.e at 2024-04-25T01:24:50.718 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 bash[62227]: 6ffe41c2eb68c3f63b990b221c57b59916bbd003808cb8b009bdc666ebaa0a3b 2024-04-25T01:24:50.719 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:24:50 smithi038 systemd[1]: Started Ceph mds.e for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:51.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:51.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:51.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 -- 172.21.15.16:0/3751482301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 msgr2=0x7f8f481008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/3751482301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f481008e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009900 tx=0x7f8f3802a410 comp rx=0 tx=0).stop 2024-04-25T01:24:51.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 -- 172.21.15.16:0/3751482301 shutdown_connections 2024-04-25T01:24:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/3751482301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48102070 0x7f8f48108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/3751482301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f481016b0 0x7f8f48101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/3751482301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f481008e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 -- 172.21.15.16:0/3751482301 >> 172.21.15.16:0/3751482301 conn(0x7f8f480fba50 msgr2=0x7f8f480fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 -- 172.21.15.16:0/3751482301 shutdown_connections 2024-04-25T01:24:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.171+0000 7f8f4d39e640 1 -- 172.21.15.16:0/3751482301 wait complete. 2024-04-25T01:24:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 Processor -- start 2024-04-25T01:24:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 -- start start 2024-04-25T01:24:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f48197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f481016b0 0x7f8f48197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48102070 0x7f8f4819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4810c050 con 0x7f8f481004c0 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4810bed0 con 0x7f8f481016b0 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f4d39e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4810c1d0 con 0x7f8f48102070 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f48197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f47fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f48197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39910/0 (socket says 172.21.15.16:39910) 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.172+0000 7f8f47fff640 1 -- 172.21.15.16:0/1050664230 learned_addr learned my addr 172.21.15.16:0/1050664230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f4cb9d640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48102070 0x7f8f4819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f477fe640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f481016b0 0x7f8f48197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f47fff640 1 -- 172.21.15.16:0/1050664230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48102070 msgr2=0x7f8f4819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f47fff640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48102070 0x7f8f4819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f47fff640 1 -- 172.21.15.16:0/1050664230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f481016b0 msgr2=0x7f8f48197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f47fff640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f481016b0 0x7f8f48197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f47fff640 1 -- 172.21.15.16:0/1050664230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4819f450 con 0x7f8f481004c0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f47fff640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f48197830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f380314e0 tx=0x7f8f38003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3803f070 con 0x7f8f481004c0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f4819f6e0 con 0x7f8f481004c0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f38031e40 con 0x7f8f481004c0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38035030 con 0x7f8f481004c0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.173+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f4819fbd0 con 0x7f8f481004c0 2024-04-25T01:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.174+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f4819fe30 con 0x7f8f481004c0 2024-04-25T01:24:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.175+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8f38042070 con 0x7f8f481004c0 2024-04-25T01:24:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.176+0000 7f8f457fa640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f300a2b90 0x7f8f300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.176+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8f380ef550 con 0x7f8f481004c0 2024-04-25T01:24:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.176+0000 7f8f477fe640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f300a2b90 0x7f8f300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.176+0000 7f8f477fe640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f300a2b90 0x7f8f300a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8f48198d40 tx=0x7f8f3c00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.180+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8f380bc950 con 0x7f8f481004c0 2024-04-25T01:24:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.404+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f481a0000 con 0x7f8f481004c0 2024-04-25T01:24:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.405+0000 7f8f457fa640 1 -- 172.21.15.16:0/1050664230 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f8f380bc250 con 0x7f8f481004c0 2024-04-25T01:24:51.385 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:51.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T01:24:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f300a2b90 msgr2=0x7f8f300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f300a2b90 0x7f8f300a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8f48198d40 tx=0x7f8f3c00a4c0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 msgr2=0x7f8f48197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f48197830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f380314e0 tx=0x7f8f38003c80 comp rx=0 tx=0).stop 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 shutdown_connections 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f300a2b90 0x7f8f300a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48102070 0x7f8f4819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f481016b0 0x7f8f48197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 --2- 172.21.15.16:0/1050664230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f481004c0 0x7f8f48197830 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 >> 172.21.15.16:0/1050664230 conn(0x7f8f480fba50 msgr2=0x7f8f48069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 shutdown_connections 2024-04-25T01:24:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:51.407+0000 7f8f4d39e640 1 -- 172.21.15.16:0/1050664230 wait complete. 2024-04-25T01:24:51.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:51 smithi016 ceph-mon[29153]: Deploying daemon mds.f on smithi156 2024-04-25T01:24:51.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:51 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6834/2628472521,v1:172.21.15.38:6835/2628472521] up:boot 2024-04-25T01:24:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:51 smithi016 ceph-mon[29153]: fsmap 5 up:standby 2024-04-25T01:24:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:24:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:51 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1050664230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:51.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:51 smithi156 ceph-mon[32631]: Deploying daemon mds.f on smithi156 2024-04-25T01:24:51.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:51 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6834/2628472521,v1:172.21.15.38:6835/2628472521] up:boot 2024-04-25T01:24:51.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:51 smithi156 ceph-mon[32631]: fsmap 5 up:standby 2024-04-25T01:24:51.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:24:51.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:51 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1050664230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:51.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:51 smithi038 ceph-mon[32841]: Deploying daemon mds.f on smithi156 2024-04-25T01:24:51.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:51 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6834/2628472521,v1:172.21.15.38:6835/2628472521] up:boot 2024-04-25T01:24:51.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:51 smithi038 ceph-mon[32841]: fsmap 5 up:standby 2024-04-25T01:24:51.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:51 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:24:51.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:51 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1050664230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:52.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:52 smithi156 ceph-mon[32631]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:52.751 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:52 smithi156 systemd[1]: Starting Ceph mds.f for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:52.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:52 smithi038 ceph-mon[32841]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:53.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:52 smithi016 ceph-mon[29153]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:53.150 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:52 smithi156 podman[63300]: 2024-04-25 01:24:52.813996812 +0000 UTC m=+0.018377406 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:53.150 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:52 smithi156 podman[63300]: 2024-04-25 01:24:52.934311579 +0000 UTC m=+0.138692199 container create 554da208df0b26296c0efce77ffc6bc4a4f41871389ddda54dfe869d82b72b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:24:53.150 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:53 smithi156 podman[63300]: 2024-04-25 01:24:53.170760299 +0000 UTC m=+0.375140877 container init 554da208df0b26296c0efce77ffc6bc4a4f41871389ddda54dfe869d82b72b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T01:24:53.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:53.501 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:53 smithi156 podman[63300]: 2024-04-25 01:24:53.173484302 +0000 UTC m=+0.377864884 container start 554da208df0b26296c0efce77ffc6bc4a4f41871389ddda54dfe869d82b72b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T01:24:53.501 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:53 smithi156 ceph-mds[63318]: -- 172.21.15.156:0/2761484357 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5597149cba40 con 0x55971499ec00 2024-04-25T01:24:53.501 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f[63314]: starting mds.f at 2024-04-25T01:24:53.501 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:53 smithi156 bash[63300]: 554da208df0b26296c0efce77ffc6bc4a4f41871389ddda54dfe869d82b72b21 2024-04-25T01:24:53.501 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:24:53 smithi156 systemd[1]: Started Ceph mds.f for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 -- 172.21.15.16:0/2187577332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c1006a0 msgr2=0x7efd8c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 --2- 172.21.15.16:0/2187577332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c1006a0 0x7efd8c100ac0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009900 tx=0x7efd7c02a410 comp rx=0 tx=0).stop 2024-04-25T01:24:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 -- 172.21.15.16:0/2187577332 shutdown_connections 2024-04-25T01:24:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 --2- 172.21.15.16:0/2187577332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c102250 0x7efd8c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 --2- 172.21.15.16:0/2187577332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c101890 0x7efd8c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 --2- 172.21.15.16:0/2187577332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c1006a0 0x7efd8c100ac0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 -- 172.21.15.16:0/2187577332 >> 172.21.15.16:0/2187577332 conn(0x7efd8c0fbc30 msgr2=0x7efd8c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 -- 172.21.15.16:0/2187577332 shutdown_connections 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 -- 172.21.15.16:0/2187577332 wait complete. 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 Processor -- start 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.532+0000 7efd942cd640 1 -- start start 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 0x7efd8c104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 0x7efd8c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c102250 0x7efd8c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c10c1d0 con 0x7efd8c101890 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c10c050 con 0x7efd8c102250 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c10c350 con 0x7efd8c1006a0 2024-04-25T01:24:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 0x7efd8c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd92069640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 0x7efd8c104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 0x7efd8c102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46466/0 (socket says 172.21.15.16:46466) 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd92069640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 0x7efd8c104590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:47266/0 (socket says 172.21.15.16:47266) 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 -- 172.21.15.16:0/3185091741 learned_addr learned my addr 172.21.15.16:0/3185091741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd9286a640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c102250 0x7efd8c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 -- 172.21.15.16:0/3185091741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 msgr2=0x7efd8c104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 0x7efd8c104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 -- 172.21.15.16:0/3185091741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c102250 msgr2=0x7efd8c1031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c102250 0x7efd8c1031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 -- 172.21.15.16:0/3185091741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c1039b0 con 0x7efd8c101890 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd9286a640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c102250 0x7efd8c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd92069640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 0x7efd8c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd91868640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 0x7efd8c102c70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efd8800c890 tx=0x7efd8800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:53.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88004220 con 0x7efd8c101890 2024-04-25T01:24:53.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd880043c0 con 0x7efd8c101890 2024-04-25T01:24:53.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8c1a1b20 con 0x7efd8c101890 2024-04-25T01:24:53.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.533+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88004a00 con 0x7efd8c101890 2024-04-25T01:24:53.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.534+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8c1a2040 con 0x7efd8c101890 2024-04-25T01:24:53.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.534+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8c1a2270 con 0x7efd8c101890 2024-04-25T01:24:53.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.536+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efd88004ba0 con 0x7efd8c101890 2024-04-25T01:24:53.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.536+0000 7efd837fe640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd780a2b90 0x7efd780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:53.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.536+0000 7efd92069640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd780a2b90 0x7efd780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:53.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.536+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efd880cec70 con 0x7efd8c101890 2024-04-25T01:24:53.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.537+0000 7efd92069640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd780a2b90 0x7efd780a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efd7c004770 tx=0x7efd7c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:53.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.540+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efd88014030 con 0x7efd8c101890 2024-04-25T01:24:53.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.765+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd8c1a2560 con 0x7efd8c101890 2024-04-25T01:24:53.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.766+0000 7efd837fe640 1 -- 172.21.15.16:0/3185091741 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7efd8809c070 con 0x7efd8c101890 2024-04-25T01:24:53.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:53.747 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T01:24:53.747 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2024-04-25T01:24:53.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd780a2b90 msgr2=0x7efd780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd780a2b90 0x7efd780a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efd7c004770 tx=0x7efd7c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 msgr2=0x7efd8c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 0x7efd8c102c70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efd8800c890 tx=0x7efd8800cd50 comp rx=0 tx=0).stop 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 shutdown_connections 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd780a2b90 0x7efd780a5050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd8c102250 0x7efd8c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd8c101890 0x7efd8c102c70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 --2- 172.21.15.16:0/3185091741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd8c1006a0 0x7efd8c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 >> 172.21.15.16:0/3185091741 conn(0x7efd8c0fbc30 msgr2=0x7efd8c069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 shutdown_connections 2024-04-25T01:24:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:53.768+0000 7efd942cd640 1 -- 172.21.15.16:0/3185091741 wait complete. 2024-04-25T01:24:54.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: Deploying daemon mds.c on smithi156 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6834/2186765223,v1:172.21.15.156:6835/2186765223] up:boot 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: fsmap 6 up:standby 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:24:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:54 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3185091741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:54.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: Deploying daemon mds.c on smithi156 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6834/2186765223,v1:172.21.15.156:6835/2186765223] up:boot 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: fsmap 6 up:standby 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:24:54.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:54 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3185091741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: Deploying daemon mds.c on smithi156 2024-04-25T01:24:54.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6834/2186765223,v1:172.21.15.156:6835/2186765223] up:boot 2024-04-25T01:24:54.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: fsmap 6 up:standby 2024-04-25T01:24:54.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:24:54.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:54 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3185091741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:55.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:55 smithi016 ceph-mon[29153]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:55.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:55 smithi038 ceph-mon[32841]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:55.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:55 smithi156 ceph-mon[32631]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:55.751 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:55 smithi156 systemd[1]: Starting Ceph mds.c for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:55.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 -- 172.21.15.16:0/3692945679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 msgr2=0x7f66e4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 --2- 172.21.15.16:0/3692945679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4106f20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009900 tx=0x7f66cc02a410 comp rx=0 tx=0).stop 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 -- 172.21.15.16:0/3692945679 shutdown_connections 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 --2- 172.21.15.16:0/3692945679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4107460 0x7f66e41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 --2- 172.21.15.16:0/3692945679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4106f20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 --2- 172.21.15.16:0/3692945679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 0x7f66e41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 -- 172.21.15.16:0/3692945679 >> 172.21.15.16:0/3692945679 conn(0x7f66e40fbbf0 msgr2=0x7f66e40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 -- 172.21.15.16:0/3692945679 shutdown_connections 2024-04-25T01:24:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 -- 172.21.15.16:0/3692945679 wait complete. 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.895+0000 7f66ea128640 1 Processor -- start 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 -- start start 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 0x7f66e4197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4107460 0x7f66e419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410d2b0 con 0x7f66e4104b30 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410d130 con 0x7f66e4102200 2024-04-25T01:24:55.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66ea128640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410d430 con 0x7f66e4107460 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46478/0 (socket says 172.21.15.16:46478) 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 -- 172.21.15.16:0/3354985553 learned_addr learned my addr 172.21.15.16:0/3354985553 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e3fff640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4107460 0x7f66e419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e37fe640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 0x7f66e4197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 -- 172.21.15.16:0/3354985553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4107460 msgr2=0x7f66e419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4107460 0x7f66e419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 -- 172.21.15.16:0/3354985553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 msgr2=0x7f66e4197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 0x7f66e4197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 -- 172.21.15.16:0/3354985553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e419f5a0 con 0x7f66e4104b30 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e37fe640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 0x7f66e4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.896+0000 7f66e2ffd640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4197ec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0314e0 tx=0x7f66cc031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.897+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc03f030 con 0x7f66e4104b30 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.897+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66cc0044b0 con 0x7f66e4104b30 2024-04-25T01:24:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.897+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e419f830 con 0x7f66e4104b30 2024-04-25T01:24:55.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.897+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc035070 con 0x7f66e4104b30 2024-04-25T01:24:55.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.897+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e419fd20 con 0x7f66e4104b30 2024-04-25T01:24:55.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.898+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e40fd6e0 con 0x7f66e4104b30 2024-04-25T01:24:55.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.899+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f66cc042030 con 0x7f66e4104b30 2024-04-25T01:24:55.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.900+0000 7f66e0ff9640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66d00a2b90 0x7f66d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.900+0000 7f66e37fe640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66d00a2b90 0x7f66d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.900+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f66cc0ef580 con 0x7f66e4104b30 2024-04-25T01:24:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.900+0000 7f66e37fe640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66d00a2b90 0x7f66d00a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66d4007e70 tx=0x7f66d4007850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:55.903+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f66cc0bc8e0 con 0x7f66e4104b30 2024-04-25T01:24:56.056 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:55 smithi156 podman[63607]: 2024-04-25 01:24:55.721820135 +0000 UTC m=+0.017891469 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:56.057 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:55 smithi156 podman[63607]: 2024-04-25 01:24:55.848666784 +0000 UTC m=+0.144738101 container create 9efd5d18a612b8340f6e40068343d4787e5bc67f45fe8f082b34b75ac330e360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-25T01:24:56.057 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:56 smithi156 podman[63607]: 2024-04-25 01:24:56.077136773 +0000 UTC m=+0.373208089 container init 9efd5d18a612b8340f6e40068343d4787e5bc67f45fe8f082b34b75ac330e360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T01:24:56.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.127+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66e402d170 con 0x7f66e4104b30 2024-04-25T01:24:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.128+0000 7f66e0ff9640 1 -- 172.21.15.16:0/3354985553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f66cc0dae20 con 0x7f66e4104b30 2024-04-25T01:24:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:56.108 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T01:24:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66d00a2b90 msgr2=0x7f66d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66d00a2b90 0x7f66d00a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66d4007e70 tx=0x7f66d4007850 comp rx=0 tx=0).stop 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 msgr2=0x7f66e4197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4197ec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0314e0 tx=0x7f66cc031a30 comp rx=0 tx=0).stop 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 shutdown_connections 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66d00a2b90 0x7f66d00a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4107460 0x7f66e419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66e4104b30 0x7f66e4197ec0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 --2- 172.21.15.16:0/3354985553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4102200 0x7f66e4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 >> 172.21.15.16:0/3354985553 conn(0x7f66e40fbbf0 msgr2=0x7f66e4102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 shutdown_connections 2024-04-25T01:24:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:56.130+0000 7f66ea128640 1 -- 172.21.15.16:0/3354985553 wait complete. 2024-04-25T01:24:56.355 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:56 smithi156 podman[63607]: 2024-04-25 01:24:56.080028715 +0000 UTC m=+0.376100032 container start 9efd5d18a612b8340f6e40068343d4787e5bc67f45fe8f082b34b75ac330e360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T01:24:56.355 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mds[63626]: -- 172.21.15.156:0/2875127055 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ea5df6da40 con 0x55ea5df40800 2024-04-25T01:24:56.355 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c[63622]: starting mds.c at 2024-04-25T01:24:56.355 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:56 smithi156 bash[63607]: 9efd5d18a612b8340f6e40068343d4787e5bc67f45fe8f082b34b75ac330e360 2024-04-25T01:24:56.355 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:24:56 smithi156 systemd[1]: Started Ceph mds.c for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:56.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3354985553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:56.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:56 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:56.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3354985553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:56.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:56.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:56.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:56 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:56.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3354985553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:56.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.753 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:56.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:56.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:56.754 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:56 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:57.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:57 smithi016 ceph-mon[29153]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:57.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:57 smithi016 ceph-mon[29153]: Deploying daemon mds.h on smithi038 2024-04-25T01:24:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:57 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6836/2504948433,v1:172.21.15.156:6837/2504948433] up:boot 2024-04-25T01:24:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:57 smithi016 ceph-mon[29153]: fsmap 7 up:standby 2024-04-25T01:24:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:24:57.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:57 smithi038 ceph-mon[32841]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:57.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:57 smithi038 ceph-mon[32841]: Deploying daemon mds.h on smithi038 2024-04-25T01:24:57.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:57 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6836/2504948433,v1:172.21.15.156:6837/2504948433] up:boot 2024-04-25T01:24:57.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:57 smithi038 ceph-mon[32841]: fsmap 7 up:standby 2024-04-25T01:24:57.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:24:57.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:57 smithi156 ceph-mon[32631]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:57.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:57 smithi156 ceph-mon[32631]: Deploying daemon mds.h on smithi038 2024-04-25T01:24:57.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:57 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6836/2504948433,v1:172.21.15.156:6837/2504948433] up:boot 2024-04-25T01:24:57.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:57 smithi156 ceph-mon[32631]: fsmap 7 up:standby 2024-04-25T01:24:57.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:24:58.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:24:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 -- 172.21.15.16:0/2906055556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea81006a0 msgr2=0x7f3ea8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 --2- 172.21.15.16:0/2906055556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea81006a0 0x7f3ea8100ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009900 tx=0x7f3e9002a410 comp rx=0 tx=0).stop 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 -- 172.21.15.16:0/2906055556 shutdown_connections 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 --2- 172.21.15.16:0/2906055556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea8102250 0x7f3ea8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 --2- 172.21.15.16:0/2906055556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea8101890 0x7f3ea8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 --2- 172.21.15.16:0/2906055556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea81006a0 0x7f3ea8100ac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 -- 172.21.15.16:0/2906055556 >> 172.21.15.16:0/2906055556 conn(0x7f3ea80fbc30 msgr2=0x7f3ea80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 -- 172.21.15.16:0/2906055556 shutdown_connections 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.257+0000 7f3eae389640 1 -- 172.21.15.16:0/2906055556 wait complete. 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 Processor -- start 2024-04-25T01:24:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 -- start start 2024-04-25T01:24:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea81006a0 0x7f3ea8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea8101890 0x7f3ea8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 0x7f3ea819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea810d310 con 0x7f3ea8102250 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea810d190 con 0x7f3ea81006a0 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eae389640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea810d490 con 0x7f3ea8101890 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eac926640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 0x7f3ea819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3ea7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea81006a0 0x7f3ea8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eac926640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 0x7f3ea819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46500/0 (socket says 172.21.15.16:46500) 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3eac926640 1 -- 172.21.15.16:0/2447706168 learned_addr learned my addr 172.21.15.16:0/2447706168 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.258+0000 7f3ea77fe640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea8101890 0x7f3ea8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eac926640 1 -- 172.21.15.16:0/2447706168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea8101890 msgr2=0x7f3ea8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eac926640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea8101890 0x7f3ea8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eac926640 1 -- 172.21.15.16:0/2447706168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea81006a0 msgr2=0x7f3ea8197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eac926640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea81006a0 0x7f3ea8197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eac926640 1 -- 172.21.15.16:0/2447706168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea819f640 con 0x7f3ea8102250 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3ea7fff640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea81006a0 0x7f3ea8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eac926640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 0x7f3ea819efd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000eb10 tx=0x7f3ea000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004b20 con 0x7f3ea8102250 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ea0004cc0 con 0x7f3ea8102250 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea819f930 con 0x7f3ea8102250 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0010690 con 0x7f3ea8102250 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.259+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea819fe30 con 0x7f3ea8102250 2024-04-25T01:24:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.260+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea8107b30 con 0x7f3ea8102250 2024-04-25T01:24:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.261+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3ea0005020 con 0x7f3ea8102250 2024-04-25T01:24:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.262+0000 7f3ea57fa640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3e940a2b90 0x7f3e940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.262+0000 7f3ea7fff640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3e940a2b90 0x7f3e940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.262+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3ea00cf1c0 con 0x7f3ea8102250 2024-04-25T01:24:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.262+0000 7f3ea7fff640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3e940a2b90 0x7f3e940a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3e90005ab0 tx=0x7f3e90005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.267+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3ea009c5c0 con 0x7f3ea8102250 2024-04-25T01:24:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.489+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ea802d170 con 0x7f3ea8102250 2024-04-25T01:24:58.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.490+0000 7f3ea57fa640 1 -- 172.21.15.16:0/2447706168 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f3ea009bec0 con 0x7f3ea8102250 2024-04-25T01:24:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:24:58.471 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:24:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-25T01:24:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3e940a2b90 msgr2=0x7f3e940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3e940a2b90 0x7f3e940a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3e90005ab0 tx=0x7f3e90005a20 comp rx=0 tx=0).stop 2024-04-25T01:24:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 msgr2=0x7f3ea819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 0x7f3ea819efd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000eb10 tx=0x7f3ea000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 shutdown_connections 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3e940a2b90 0x7f3e940a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ea8102250 0x7f3ea819efd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea8101890 0x7f3ea8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 --2- 172.21.15.16:0/2447706168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ea81006a0 0x7f3ea8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 >> 172.21.15.16:0/2447706168 conn(0x7f3ea80fbc30 msgr2=0x7f3ea8104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 shutdown_connections 2024-04-25T01:24:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:24:58.492+0000 7f3eae389640 1 -- 172.21.15.16:0/2447706168 wait complete. 2024-04-25T01:24:58.716 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:58 smithi038 systemd[1]: Starting Ceph mds.h for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:24:58.988 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:58 smithi038 podman[62540]: 2024-04-25 01:24:58.650679906 +0000 UTC m=+0.018703813 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:58.988 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:58 smithi038 podman[62540]: 2024-04-25 01:24:58.76496232 +0000 UTC m=+0.132986250 container create 0bc659dae5f40df83c01be0310090c79a85aed9f3769d78cef48a57dcb7ace0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T01:24:58.989 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:59 smithi038 podman[62540]: 2024-04-25 01:24:59.009501038 +0000 UTC m=+0.377524927 container init 0bc659dae5f40df83c01be0310090c79a85aed9f3769d78cef48a57dcb7ace0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:24:59.360 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:59 smithi038 podman[62540]: 2024-04-25 01:24:59.012307767 +0000 UTC m=+0.380331657 container start 0bc659dae5f40df83c01be0310090c79a85aed9f3769d78cef48a57dcb7ace0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T01:24:59.360 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mds[62559]: -- 172.21.15.38:0/3779347652 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562cd28d7a40 con 0x562cd28ab000 2024-04-25T01:24:59.360 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h[62555]: starting mds.h at 2024-04-25T01:24:59.360 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:59 smithi038 bash[62540]: 0bc659dae5f40df83c01be0310090c79a85aed9f3769d78cef48a57dcb7ace0d 2024-04-25T01:24:59.360 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:24:59 smithi038 systemd[1]: Started Ceph mds.h for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:24:59.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:59.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2447706168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:24:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:59.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:59.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2447706168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:59.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:24:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:59.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2447706168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:24:59.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:24:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:00.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 -- 172.21.15.16:0/2755117263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc1022c0 msgr2=0x7f25dc1046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 --2- 172.21.15.16:0/2755117263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc1022c0 0x7f25dc1046b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009900 tx=0x7f25d802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 -- 172.21.15.16:0/2755117263 shutdown_connections 2024-04-25T01:25:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 --2- 172.21.15.16:0/2755117263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25dc107520 0x7f25dc109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 --2- 172.21.15.16:0/2755117263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc104bf0 0x7f25dc106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 --2- 172.21.15.16:0/2755117263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc1022c0 0x7f25dc1046b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 -- 172.21.15.16:0/2755117263 >> 172.21.15.16:0/2755117263 conn(0x7f25dc0fbc30 msgr2=0x7f25dc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 -- 172.21.15.16:0/2755117263 shutdown_connections 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.619+0000 7f25e4453640 1 -- 172.21.15.16:0/2755117263 wait complete. 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 Processor -- start 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 -- start start 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc1022c0 0x7f25dc101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 0x7f25dc100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25dc107520 0x7f25dc1007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10d330 con 0x7f25dc104bf0 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10d1b0 con 0x7f25dc1022c0 2024-04-25T01:25:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e4453640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10d4b0 con 0x7f25dc107520 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e19ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 0x7f25dc100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e19ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 0x7f25dc100270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46512/0 (socket says 172.21.15.16:46512) 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e19ee640 1 -- 172.21.15.16:0/1991570183 learned_addr learned my addr 172.21.15.16:0/1991570183 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.620+0000 7f25e29f0640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25dc107520 0x7f25dc1007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e21ef640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc1022c0 0x7f25dc101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e19ee640 1 -- 172.21.15.16:0/1991570183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25dc107520 msgr2=0x7f25dc1007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e19ee640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25dc107520 0x7f25dc1007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e19ee640 1 -- 172.21.15.16:0/1991570183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc1022c0 msgr2=0x7f25dc101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e19ee640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc1022c0 0x7f25dc101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e19ee640 1 -- 172.21.15.16:0/1991570183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc101040 con 0x7f25dc104bf0 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e21ef640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc1022c0 0x7f25dc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e19ee640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 0x7f25dc100270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00e9c0 tx=0x7f25cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00cd80 con 0x7f25dc104bf0 2024-04-25T01:25:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25dc1a1b00 con 0x7f25dc104bf0 2024-04-25T01:25:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25cc0054d0 con 0x7f25dc104bf0 2024-04-25T01:25:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc010430 con 0x7f25dc104bf0 2024-04-25T01:25:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.621+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25dc1a2050 con 0x7f25dc104bf0 2024-04-25T01:25:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.622+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25dc0fd720 con 0x7f25dc104bf0 2024-04-25T01:25:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.623+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f25cc004fd0 con 0x7f25dc104bf0 2024-04-25T01:25:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.624+0000 7f25d37fe640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25c80a2b90 0x7f25c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.624+0000 7f25e21ef640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25c80a2b90 0x7f25c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.624+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f25cc0cf1e0 con 0x7f25dc104bf0 2024-04-25T01:25:00.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.624+0000 7f25e21ef640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25c80a2b90 0x7f25c80a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f25d802a930 tx=0x7f25d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:00.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.628+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25cc09c540 con 0x7f25dc104bf0 2024-04-25T01:25:00.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:00 smithi038 ceph-mon[32841]: Deploying daemon mds.i on smithi156 2024-04-25T01:25:00.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:00 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] up:boot 2024-04-25T01:25:00.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:00 smithi038 ceph-mon[32841]: fsmap 8 up:standby 2024-04-25T01:25:00.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:25:00.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:00 smithi156 ceph-mon[32631]: Deploying daemon mds.i on smithi156 2024-04-25T01:25:00.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:00 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] up:boot 2024-04-25T01:25:00.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:00 smithi156 ceph-mon[32631]: fsmap 8 up:standby 2024-04-25T01:25:00.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:25:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.849+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25dc0623e0 con 0x7f25dc104bf0 2024-04-25T01:25:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.850+0000 7f25d37fe640 1 -- 172.21.15.16:0/1991570183 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f25cc09be40 con 0x7f25dc104bf0 2024-04-25T01:25:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:00.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T01:25:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25c80a2b90 msgr2=0x7f25c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25c80a2b90 0x7f25c80a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f25d802a930 tx=0x7f25d8005fd0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 msgr2=0x7f25dc100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 0x7f25dc100270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00e9c0 tx=0x7f25cc00ee80 comp rx=0 tx=0).stop 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 shutdown_connections 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25c80a2b90 0x7f25c80a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25dc107520 0x7f25dc1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25dc104bf0 0x7f25dc100270 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 --2- 172.21.15.16:0/1991570183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25dc1022c0 0x7f25dc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 >> 172.21.15.16:0/1991570183 conn(0x7f25dc0fbc30 msgr2=0x7f25dc102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 shutdown_connections 2024-04-25T01:25:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:00.852+0000 7f25e4453640 1 -- 172.21.15.16:0/1991570183 wait complete. 2024-04-25T01:25:00.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:00 smithi016 ceph-mon[29153]: Deploying daemon mds.i on smithi156 2024-04-25T01:25:00.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:00 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] up:boot 2024-04-25T01:25:00.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:00 smithi016 ceph-mon[29153]: fsmap 8 up:standby 2024-04-25T01:25:00.842 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:25:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:01 smithi156 ceph-mon[32631]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:01 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1991570183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:01.501 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 systemd[1]: Starting Ceph mds.i for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:25:01.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:01 smithi038 ceph-mon[32841]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:01.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:01 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1991570183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:01.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:01 smithi016 ceph-mon[29153]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:01.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:01 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1991570183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:01.878 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 podman[63915]: 2024-04-25 01:25:01.529779656 +0000 UTC m=+0.018115805 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:25:01.878 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 podman[63915]: 2024-04-25 01:25:01.646029393 +0000 UTC m=+0.134365513 container create 492b7fcb72ca84f94dd0a8c40baf14acf3cce8bdf8e56739332e41fd565705db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:25:02.251 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 podman[63915]: 2024-04-25 01:25:01.898989312 +0000 UTC m=+0.387325435 container init 492b7fcb72ca84f94dd0a8c40baf14acf3cce8bdf8e56739332e41fd565705db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T01:25:02.251 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 podman[63915]: 2024-04-25 01:25:01.901644424 +0000 UTC m=+0.389980555 container start 492b7fcb72ca84f94dd0a8c40baf14acf3cce8bdf8e56739332e41fd565705db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:25:02.251 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 ceph-mds[63934]: -- 172.21.15.156:0/1773559326 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb63af1a40 con 0x55cb63ac4c00 2024-04-25T01:25:02.251 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i[63930]: starting mds.i at 2024-04-25T01:25:02.251 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 bash[63915]: 492b7fcb72ca84f94dd0a8c40baf14acf3cce8bdf8e56739332e41fd565705db 2024-04-25T01:25:02.251 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:25:01 smithi156 systemd[1]: Started Ceph mds.i for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:25:02.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:02.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 -- 172.21.15.16:0/1808928851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c102230 msgr2=0x7f651c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 --2- 172.21.15.16:0/1808928851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c102230 0x7f651c108ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6514009ca0 tx=0x7f651402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 -- 172.21.15.16:0/1808928851 shutdown_connections 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 --2- 172.21.15.16:0/1808928851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c102230 0x7f651c108ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 --2- 172.21.15.16:0/1808928851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f651c101870 0x7f651c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 --2- 172.21.15.16:0/1808928851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c100680 0x7f651c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 -- 172.21.15.16:0/1808928851 >> 172.21.15.16:0/1808928851 conn(0x7f651c0fbbf0 msgr2=0x7f651c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 -- 172.21.15.16:0/1808928851 shutdown_connections 2024-04-25T01:25:02.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.969+0000 7f6520b14640 1 -- 172.21.15.16:0/1808928851 wait complete. 2024-04-25T01:25:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 Processor -- start 2024-04-25T01:25:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 -- start start 2024-04-25T01:25:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c100680 0x7f651c0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:02.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f651c101870 0x7f651c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 0x7f651c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10c270 con 0x7f651c102230 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10c0f0 con 0x7f651c100680 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.970+0000 7f6520b14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10c3f0 con 0x7f651c101870 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 0x7f651c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 0x7f651c19ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49138/0 (socket says 172.21.15.16:49138) 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 -- 172.21.15.16:0/633007553 learned_addr learned my addr 172.21.15.16:0/633007553 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651a59c640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c100680 0x7f651c0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f6519d9b640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f651c101870 0x7f651c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 -- 172.21.15.16:0/633007553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f651c101870 msgr2=0x7f651c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f651c101870 0x7f651c19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 -- 172.21.15.16:0/633007553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c100680 msgr2=0x7f651c0694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:02.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c100680 0x7f651c0694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 -- 172.21.15.16:0/633007553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c19f630 con 0x7f651c102230 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f651ad9d640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 0x7f651c19ed70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6514031040 tx=0x7f6514003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514032ec0 con 0x7f651c102230 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65140044f0 con 0x7f651c102230 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c19f8c0 con 0x7f651c102230 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514030bd0 con 0x7f651c102230 2024-04-25T01:25:02.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.971+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c19fe10 con 0x7f651c102230 2024-04-25T01:25:02.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.973+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651c1079a0 con 0x7f651c102230 2024-04-25T01:25:02.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.974+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6514004040 con 0x7f651c102230 2024-04-25T01:25:02.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.974+0000 7f650b7fe640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f65040a2d30 0x7f65040a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:02.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.975+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f65140f0250 con 0x7f651c102230 2024-04-25T01:25:02.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.975+0000 7f651a59c640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f65040a2d30 0x7f65040a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:02.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.975+0000 7f651a59c640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f65040a2d30 0x7f65040a51f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f651c1016d0 tx=0x7f650c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:02.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:02.979+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f65140bd5b0 con 0x7f651c102230 2024-04-25T01:25:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.206+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f651c062340 con 0x7f651c102230 2024-04-25T01:25:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.207+0000 7f650b7fe640 1 -- 172.21.15.16:0/633007553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f65140bceb0 con 0x7f651c102230 2024-04-25T01:25:03.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:03.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:25:03.188 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-04-25T01:25:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f65040a2d30 msgr2=0x7f65040a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f65040a2d30 0x7f65040a51f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f651c1016d0 tx=0x7f650c009110 comp rx=0 tx=0).stop 2024-04-25T01:25:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 msgr2=0x7f651c19ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 0x7f651c19ed70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6514031040 tx=0x7f6514003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 shutdown_connections 2024-04-25T01:25:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f65040a2d30 0x7f65040a51f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c102230 0x7f651c19ed70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f651c101870 0x7f651c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 --2- 172.21.15.16:0/633007553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f651c100680 0x7f651c0694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 >> 172.21.15.16:0/633007553 conn(0x7f651c0fbbf0 msgr2=0x7f651c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 shutdown_connections 2024-04-25T01:25:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:03.210+0000 7f6520b14640 1 -- 172.21.15.16:0/633007553 wait complete. 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:03.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:03.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: Deploying daemon mds.g on smithi016 2024-04-25T01:25:03.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] up:boot 2024-04-25T01:25:03.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: fsmap 9 up:standby 2024-04-25T01:25:03.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:03 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:25:03.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:03.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: Deploying daemon mds.g on smithi016 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] up:boot 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: fsmap 9 up:standby 2024-04-25T01:25:03.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:03 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: Deploying daemon mds.g on smithi016 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] up:boot 2024-04-25T01:25:03.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: fsmap 9 up:standby 2024-04-25T01:25:03.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:03 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:25:04.328 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:04 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/633007553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:04.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:04 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/633007553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:04.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:04 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/633007553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:04.610 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 podman[70224]: 2024-04-25 01:25:04.597243736 +0000 UTC m=+0.143954823 container create b451c0c1eadf12944dbe5c9bfb911d4f0209e2c0dffed3fc21d91c4044038e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T01:25:05.110 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 podman[70224]: 2024-04-25 01:25:04.867389701 +0000 UTC m=+0.414100720 container init b451c0c1eadf12944dbe5c9bfb911d4f0209e2c0dffed3fc21d91c4044038e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:25:05.110 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 podman[70224]: 2024-04-25 01:25:04.870877106 +0000 UTC m=+0.417588127 container start b451c0c1eadf12944dbe5c9bfb911d4f0209e2c0dffed3fc21d91c4044038e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:25:05.110 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 ceph-mds[70243]: -- 172.21.15.16:0/2355399799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562830721a40 con 0x5628306f5000 2024-04-25T01:25:05.111 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g[70239]: starting mds.g at 2024-04-25T01:25:05.111 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 bash[70224]: b451c0c1eadf12944dbe5c9bfb911d4f0209e2c0dffed3fc21d91c4044038e13 2024-04-25T01:25:05.111 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:25:04 smithi016 systemd[1]: Started Ceph mds.g for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:25:05.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- 172.21.15.16:0/3449825793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 msgr2=0x7fc7b4109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- 172.21.15.16:0/3449825793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b4109be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009ca0 tx=0x7fc7ac02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- 172.21.15.16:0/3449825793 shutdown_connections 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- 172.21.15.16:0/3449825793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b4109be0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- 172.21.15.16:0/3449825793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 0x7fc7b4102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- 172.21.15.16:0/3449825793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 0x7fc7b4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- 172.21.15.16:0/3449825793 >> 172.21.15.16:0/3449825793 conn(0x7fc7b40fccf0 msgr2=0x7fc7b40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- 172.21.15.16:0/3449825793 shutdown_connections 2024-04-25T01:25:05.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- 172.21.15.16:0/3449825793 wait complete. 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 Processor -- start 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- start start 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 0x7fc7b4197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 0x7fc7b4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b410d2d0 con 0x7fc7b4103350 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b410d150 con 0x7fc7b4102990 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.330+0000 7fc7badc8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b410d450 con 0x7fc7b41017a0 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49170/0 (socket says 172.21.15.16:49170) 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 -- 172.21.15.16:0/554107684 learned_addr learned my addr 172.21.15.16:0/554107684 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:05.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b3fff640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 0x7fc7b4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b8b64640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 0x7fc7b4197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 -- 172.21.15.16:0/554107684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 msgr2=0x7fc7b4197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 0x7fc7b4197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 -- 172.21.15.16:0/554107684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 msgr2=0x7fc7b4197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 0x7fc7b4197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 -- 172.21.15.16:0/554107684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b419f590 con 0x7fc7b4103350 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b3fff640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 0x7fc7b4197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b8b64640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 0x7fc7b4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b9365640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b419ef20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009dc0 tx=0x7fc7ac003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac032ec0 con 0x7fc7b4103350 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7ac0044f0 con 0x7fc7b4103350 2024-04-25T01:25:05.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b419f820 con 0x7fc7b4103350 2024-04-25T01:25:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.331+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac030d40 con 0x7fc7b4103350 2024-04-25T01:25:05.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.332+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b419fd70 con 0x7fc7b4103350 2024-04-25T01:25:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.333+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b4108b40 con 0x7fc7b4103350 2024-04-25T01:25:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.334+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc7ac004040 con 0x7fc7b4103350 2024-04-25T01:25:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.335+0000 7fc7b1ffb640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc7a40a2d30 0x7fc7a40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.335+0000 7fc7b8b64640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc7a40a2d30 0x7fc7a40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.335+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc7ac0f0300 con 0x7fc7b4103350 2024-04-25T01:25:05.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.336+0000 7fc7b8b64640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc7a40a2d30 0x7fc7a40a51f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80058c0 tx=0x7fc7a8005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:05.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.340+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc7ac0bd660 con 0x7fc7b4103350 2024-04-25T01:25:05.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:05.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:05.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:05.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:05 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:05.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:05 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.580+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7b402d170 con 0x7fc7b4103350 2024-04-25T01:25:05.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.580+0000 7fc7b1ffb640 1 -- 172.21.15.16:0/554107684 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fc7ac0bcf60 con 0x7fc7b4103350 2024-04-25T01:25:05.561 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:05.561 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:25:05.561 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc7a40a2d30 msgr2=0x7fc7a40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc7a40a2d30 0x7fc7a40a51f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80058c0 tx=0x7fc7a8005ec0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 msgr2=0x7fc7b419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b419ef20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009dc0 tx=0x7fc7ac003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 shutdown_connections 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc7a40a2d30 0x7fc7a40a51f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7b4103350 0x7fc7b419ef20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7b4102990 0x7fc7b4197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 --2- 172.21.15.16:0/554107684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7b41017a0 0x7fc7b4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:05.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 >> 172.21.15.16:0/554107684 conn(0x7fc7b40fccf0 msgr2=0x7fc7b4105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:05.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 shutdown_connections 2024-04-25T01:25:05.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:05.583+0000 7fc7badc8640 1 -- 172.21.15.16:0/554107684 wait complete. 2024-04-25T01:25:05.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:05.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:05.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:05.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:05.574 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:05 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:06.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:06 smithi038 ceph-mon[32841]: Deploying daemon mds.a on smithi016 2024-04-25T01:25:06.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:06 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.16:6838/4015716085,v1:172.21.15.16:6839/4015716085] up:boot 2024-04-25T01:25:06.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:06 smithi038 ceph-mon[32841]: fsmap 10 up:standby 2024-04-25T01:25:06.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:06 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:25:06.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:06 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/554107684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:06.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:06 smithi156 ceph-mon[32631]: Deploying daemon mds.a on smithi016 2024-04-25T01:25:06.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:06 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.16:6838/4015716085,v1:172.21.15.16:6839/4015716085] up:boot 2024-04-25T01:25:06.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:06 smithi156 ceph-mon[32631]: fsmap 10 up:standby 2024-04-25T01:25:06.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:06 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:25:06.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:06 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/554107684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:06.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:06 smithi016 ceph-mon[29153]: Deploying daemon mds.a on smithi016 2024-04-25T01:25:06.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:06 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.16:6838/4015716085,v1:172.21.15.16:6839/4015716085] up:boot 2024-04-25T01:25:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:06 smithi016 ceph-mon[29153]: fsmap 10 up:standby 2024-04-25T01:25:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:06 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:25:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:06 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/554107684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:07.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:07 smithi016 ceph-mon[29153]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:07.361 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 systemd[1]: Starting Ceph mds.a for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:25:07.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:07 smithi038 ceph-mon[32841]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:07.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:07 smithi156 ceph-mon[32631]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:07.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 -- 172.21.15.16:0/108747752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 msgr2=0x7fdff4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 --2- 172.21.15.16:0/108747752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff4108ae0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009ca0 tx=0x7fdfec02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 -- 172.21.15.16:0/108747752 shutdown_connections 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 --2- 172.21.15.16:0/108747752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff4108ae0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 --2- 172.21.15.16:0/108747752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff4101890 0x7fdff4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 --2- 172.21.15.16:0/108747752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdff41006a0 0x7fdff4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 -- 172.21.15.16:0/108747752 >> 172.21.15.16:0/108747752 conn(0x7fdff40fbc30 msgr2=0x7fdff40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 -- 172.21.15.16:0/108747752 shutdown_connections 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.703+0000 7fdffac12640 1 -- 172.21.15.16:0/108747752 wait complete. 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 Processor -- start 2024-04-25T01:25:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 -- start start 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff41006a0 0x7fdff41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdff4101890 0x7fdff4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdff410c1d0 con 0x7fdff4102250 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdff410c050 con 0x7fdff41006a0 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdffac12640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdff410c350 con 0x7fdff4101890 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdff91af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdff91af640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49196/0 (socket says 172.21.15.16:49196) 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdff91af640 1 -- 172.21.15.16:0/3027777913 learned_addr learned my addr 172.21.15.16:0/3027777913 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdff3fff640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdff4101890 0x7fdff4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.704+0000 7fdff89ae640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff41006a0 0x7fdff41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff91af640 1 -- 172.21.15.16:0/3027777913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdff4101890 msgr2=0x7fdff4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff91af640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdff4101890 0x7fdff4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff91af640 1 -- 172.21.15.16:0/3027777913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff41006a0 msgr2=0x7fdff41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff91af640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff41006a0 0x7fdff41979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff91af640 1 -- 172.21.15.16:0/3027777913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff419f580 con 0x7fdff4102250 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff91af640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff419ef10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009dc0 tx=0x7fdfec003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec032ec0 con 0x7fdff4102250 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfec0044f0 con 0x7fdff4102250 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff419f810 con 0x7fdff4102250 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec030d40 con 0x7fdff4102250 2024-04-25T01:25:07.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.705+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff419fdb0 con 0x7fdff4102250 2024-04-25T01:25:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.706+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff419ffb0 con 0x7fdff4102250 2024-04-25T01:25:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.707+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdfec004040 con 0x7fdff4102250 2024-04-25T01:25:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.708+0000 7fdff1ffb640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfe40a2db0 0x7fdfe40a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.708+0000 7fdff89ae640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfe40a2db0 0x7fdfe40a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.708+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdfec0f0100 con 0x7fdff4102250 2024-04-25T01:25:07.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.709+0000 7fdff89ae640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfe40a2db0 0x7fdfe40a5270 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8005fd0 tx=0x7fdfe8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.713+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfec0bd500 con 0x7fdff4102250 2024-04-25T01:25:07.729 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 podman[70571]: 2024-04-25 01:25:07.374739219 +0000 UTC m=+0.018770796 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:25:07.729 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 podman[70571]: 2024-04-25 01:25:07.487928503 +0000 UTC m=+0.131960080 container create 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T01:25:07.729 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 podman[70571]: 2024-04-25 01:25:07.749767822 +0000 UTC m=+0.393799399 container init 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T01:25:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.945+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdff41a0340 con 0x7fdff4102250 2024-04-25T01:25:07.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.945+0000 7fdff1ffb640 1 -- 172.21.15.16:0/3027777913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fdfec0bce00 con 0x7fdff4102250 2024-04-25T01:25:07.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:07.926 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:25:07.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-04-25T01:25:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfe40a2db0 msgr2=0x7fdfe40a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfe40a2db0 0x7fdfe40a5270 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8005fd0 tx=0x7fdfe8005850 comp rx=0 tx=0).stop 2024-04-25T01:25:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 msgr2=0x7fdff419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff419ef10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009dc0 tx=0x7fdfec003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 shutdown_connections 2024-04-25T01:25:07.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfe40a2db0 0x7fdfe40a5270 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdff4102250 0x7fdff419ef10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdff4101890 0x7fdff4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 --2- 172.21.15.16:0/3027777913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdff41006a0 0x7fdff41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 >> 172.21.15.16:0/3027777913 conn(0x7fdff40fbc30 msgr2=0x7fdff40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 shutdown_connections 2024-04-25T01:25:07.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:07.948+0000 7fdffac12640 1 -- 172.21.15.16:0/3027777913 wait complete. 2024-04-25T01:25:08.110 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 podman[70571]: 2024-04-25 01:25:07.752835984 +0000 UTC m=+0.396867564 container start 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T01:25:08.110 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 ceph-mds[70629]: -- 172.21.15.16:0/747902865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56261db53a40 con 0x56261db26c00 2024-04-25T01:25:08.110 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a[70590]: starting mds.a at 2024-04-25T01:25:08.111 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 bash[70571]: 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 2024-04-25T01:25:08.111 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:25:07 smithi016 systemd[1]: Started Ceph mds.a for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:25:09.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:09.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3027777913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: Deploying daemon mds.b on smithi038 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.16:6840/3747963498,v1:172.21.15.16:6841/3747963498] up:boot 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: fsmap 11 up:standby 2024-04-25T01:25:09.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:08 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:25:09.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:09.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3027777913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:09.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: Deploying daemon mds.b on smithi038 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.16:6840/3747963498,v1:172.21.15.16:6841/3747963498] up:boot 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: fsmap 11 up:standby 2024-04-25T01:25:09.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:08 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:25:09.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:09.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3027777913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: Deploying daemon mds.b on smithi038 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.16:6840/3747963498,v1:172.21.15.16:6841/3747963498] up:boot 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: fsmap 11 up:standby 2024-04-25T01:25:09.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:08 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:25:09.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 -- 172.21.15.16:0/869425487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301029a0 msgr2=0x7feb30102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 --2- 172.21.15.16:0/869425487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301029a0 0x7feb30102e20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb24009900 tx=0x7feb2402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 -- 172.21.15.16:0/869425487 shutdown_connections 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 --2- 172.21.15.16:0/869425487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb30103360 0x7feb30109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 --2- 172.21.15.16:0/869425487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301029a0 0x7feb30102e20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 --2- 172.21.15.16:0/869425487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301017b0 0x7feb30101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.075+0000 7feb36d5a640 1 -- 172.21.15.16:0/869425487 >> 172.21.15.16:0/869425487 conn(0x7feb300fcd40 msgr2=0x7feb300ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:10.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 -- 172.21.15.16:0/869425487 shutdown_connections 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 -- 172.21.15.16:0/869425487 wait complete. 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 Processor -- start 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 -- start start 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 0x7feb30197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301029a0 0x7feb30197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb30103360 0x7feb3019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb3010d2c0 con 0x7feb301017b0 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb3010d140 con 0x7feb301029a0 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb36d5a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb3010d440 con 0x7feb30103360 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb34af6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 0x7feb30197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb34af6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 0x7feb30197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49224/0 (socket says 172.21.15.16:49224) 2024-04-25T01:25:10.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb2ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301029a0 0x7feb30197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.076+0000 7feb34af6640 1 -- 172.21.15.16:0/3641223326 learned_addr learned my addr 172.21.15.16:0/3641223326 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb352f7640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb30103360 0x7feb3019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb34af6640 1 -- 172.21.15.16:0/3641223326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb30103360 msgr2=0x7feb3019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb34af6640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb30103360 0x7feb3019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb34af6640 1 -- 172.21.15.16:0/3641223326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301029a0 msgr2=0x7feb30197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb34af6640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301029a0 0x7feb30197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb34af6640 1 -- 172.21.15.16:0/3641223326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3019f5f0 con 0x7feb301017b0 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb2ffff640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301029a0 0x7feb30197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb34af6640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 0x7feb30197a20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feb2000e9c0 tx=0x7feb2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2000cde0 con 0x7feb301017b0 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb200108e0 con 0x7feb301017b0 2024-04-25T01:25:10.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3019f8e0 con 0x7feb301017b0 2024-04-25T01:25:10.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20018990 con 0x7feb301017b0 2024-04-25T01:25:10.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.077+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3019fe00 con 0x7feb301017b0 2024-04-25T01:25:10.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.078+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb30108bf0 con 0x7feb301017b0 2024-04-25T01:25:10.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.079+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7feb2001f020 con 0x7feb301017b0 2024-04-25T01:25:10.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.080+0000 7feb2dffb640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb180a2b90 0x7feb180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:10.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.080+0000 7feb2ffff640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb180a2b90 0x7feb180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:10.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.080+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7feb200cee10 con 0x7feb301017b0 2024-04-25T01:25:10.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.081+0000 7feb2ffff640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb180a2b90 0x7feb180a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb24006010 tx=0x7feb24000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:10.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.085+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb2009c210 con 0x7feb301017b0 2024-04-25T01:25:10.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.307+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb3002d340 con 0x7feb301017b0 2024-04-25T01:25:10.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.307+0000 7feb2dffb640 1 -- 172.21.15.16:0/3641223326 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7feb2009bb10 con 0x7feb301017b0 2024-04-25T01:25:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:10.288 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:25:10.288 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb180a2b90 msgr2=0x7feb180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb180a2b90 0x7feb180a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb24006010 tx=0x7feb24000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 msgr2=0x7feb30197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 0x7feb30197a20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feb2000e9c0 tx=0x7feb2000ee80 comp rx=0 tx=0).stop 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 shutdown_connections 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7feb180a2b90 0x7feb180a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb30103360 0x7feb3019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb301029a0 0x7feb30197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 --2- 172.21.15.16:0/3641223326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb301017b0 0x7feb30197a20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 >> 172.21.15.16:0/3641223326 conn(0x7feb300fcd40 msgr2=0x7feb30105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.309+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 shutdown_connections 2024-04-25T01:25:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:10.310+0000 7feb36d5a640 1 -- 172.21.15.16:0/3641223326 wait complete. 2024-04-25T01:25:10.466 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 systemd[1]: Starting Ceph mds.b for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:25:10.720 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 podman[62852]: 2024-04-25 01:25:10.488670584 +0000 UTC m=+0.110531306 container create 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:25:10.720 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 podman[62852]: 2024-04-25 01:25:10.396992353 +0000 UTC m=+0.018853104 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:25:10.721 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 podman[62852]: 2024-04-25 01:25:10.74154141 +0000 UTC m=+0.363402133 container init 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T01:25:11.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:10 smithi038 ceph-mon[32841]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:11.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:10 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3641223326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:11.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:10 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:11.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:10 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:11.217 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 podman[62852]: 2024-04-25 01:25:10.744289873 +0000 UTC m=+0.366150606 container start 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T01:25:11.217 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 ceph-mds[62871]: -- 172.21.15.38:0/598623461 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557bbbe33a40 con 0x557bbbe07000 2024-04-25T01:25:11.217 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: starting mds.b at 2024-04-25T01:25:11.217 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 bash[62852]: 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 2024-04-25T01:25:11.217 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:10 smithi038 systemd[1]: Started Ceph mds.b for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:25:11.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:10 smithi156 ceph-mon[32631]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:11.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:10 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3641223326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:11.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:10 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:11.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:10 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:11.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:10 smithi016 ceph-mon[29153]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:11.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:10 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3641223326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:11.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:10 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:11.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:10 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] up:boot 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: fsmap 12 up:standby 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:11 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:12.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] up:boot 2024-04-25T01:25:12.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: fsmap 12 up:standby 2024-04-25T01:25:12.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:25:12.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:11 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:12.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] up:boot 2024-04-25T01:25:12.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: fsmap 12 up:standby 2024-04-25T01:25:12.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:25:12.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:12.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:11 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:12.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 -- 172.21.15.16:0/2160670571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 msgr2=0x7ffa48100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/2160670571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa48100ac0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009900 tx=0x7ffa3802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 -- 172.21.15.16:0/2160670571 shutdown_connections 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/2160670571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa48102250 0x7ffa48108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/2160670571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48101890 0x7ffa48101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/2160670571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa48100ac0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 -- 172.21.15.16:0/2160670571 >> 172.21.15.16:0/2160670571 conn(0x7ffa480fbc30 msgr2=0x7ffa480fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 -- 172.21.15.16:0/2160670571 shutdown_connections 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 -- 172.21.15.16:0/2160670571 wait complete. 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 Processor -- start 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 -- start start 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa4806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa48101890 0x7ffa4819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.427+0000 7ffa4f0da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102250 0x7ffa4819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4f0da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c1a0 con 0x7ffa481006a0 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4f0da640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c020 con 0x7ffa48102250 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4f0da640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c320 con 0x7ffa48101890 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa4806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa4806a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37454/0 (socket says 172.21.15.16:37454) 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 -- 172.21.15.16:0/910854085 learned_addr learned my addr 172.21.15.16:0/910854085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa47fff640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa48101890 0x7ffa4819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4d677640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102250 0x7ffa4819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 -- 172.21.15.16:0/910854085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa48101890 msgr2=0x7ffa4819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa48101890 0x7ffa4819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 -- 172.21.15.16:0/910854085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102250 msgr2=0x7ffa4819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102250 0x7ffa4819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 -- 172.21.15.16:0/910854085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819f650 con 0x7ffa481006a0 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4ce76640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa4806a910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffa380314e0 tx=0x7ffa38031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3803f030 con 0x7ffa481006a0 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa380044b0 con 0x7ffa481006a0 2024-04-25T01:25:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38035070 con 0x7ffa481006a0 2024-04-25T01:25:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4819f880 con 0x7ffa481006a0 2024-04-25T01:25:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.428+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4819fd20 con 0x7ffa481006a0 2024-04-25T01:25:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.429+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa48107a90 con 0x7ffa481006a0 2024-04-25T01:25:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.430+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffa38042030 con 0x7ffa481006a0 2024-04-25T01:25:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.431+0000 7ffa45ffb640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ffa300a2b90 0x7ffa300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.431+0000 7ffa47fff640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ffa300a2b90 0x7ffa300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.431+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffa380ef0a0 con 0x7ffa481006a0 2024-04-25T01:25:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.432+0000 7ffa47fff640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ffa300a2b90 0x7ffa300a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ffa4806a680 tx=0x7ffa3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.436+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa380bbe50 con 0x7ffa481006a0 2024-04-25T01:25:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.669+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa4802d170 con 0x7ffa481006a0 2024-04-25T01:25:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.670+0000 7ffa45ffb640 1 -- 172.21.15.16:0/910854085 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7ffa380c0150 con 0x7ffa481006a0 2024-04-25T01:25:12.650 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:12.650 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T01:25:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ffa300a2b90 msgr2=0x7ffa300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ffa300a2b90 0x7ffa300a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ffa4806a680 tx=0x7ffa3c005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 msgr2=0x7ffa4806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa4806a910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffa380314e0 tx=0x7ffa38031a30 comp rx=0 tx=0).stop 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 shutdown_connections 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ffa300a2b90 0x7ffa300a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102250 0x7ffa4819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa48101890 0x7ffa4819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 --2- 172.21.15.16:0/910854085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa481006a0 0x7ffa4806a910 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 >> 172.21.15.16:0/910854085 conn(0x7ffa480fbc30 msgr2=0x7ffa48104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 shutdown_connections 2024-04-25T01:25:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.672+0000 7ffa4f0da640 1 -- 172.21.15.16:0/910854085 wait complete. 2024-04-25T01:25:12.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.792+0000 7f5978802640 1 -- 172.21.15.16:0/2008136302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 msgr2=0x7f5974101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.792+0000 7f5978802640 1 --2- 172.21.15.16:0/2008136302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f5974101bd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5964009900 tx=0x7f596402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 -- 172.21.15.16:0/2008136302 shutdown_connections 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 --2- 172.21.15.16:0/2008136302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5974103360 0x7f5974109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 --2- 172.21.15.16:0/2008136302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59741029a0 0x7f5974102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 --2- 172.21.15.16:0/2008136302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f5974101bd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 -- 172.21.15.16:0/2008136302 >> 172.21.15.16:0/2008136302 conn(0x7f59740fcd40 msgr2=0x7f59740ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 -- 172.21.15.16:0/2008136302 shutdown_connections 2024-04-25T01:25:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 -- 172.21.15.16:0/2008136302 wait complete. 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 Processor -- start 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 -- start start 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f59741979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.793+0000 7f5978802640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59741029a0 0x7f5974197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5978802640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5974103360 0x7f597419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5978802640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f597410d300 con 0x7f59741017b0 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5978802640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f597410d180 con 0x7f5974103360 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5978802640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f597410d480 con 0x7f59741029a0 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f59741979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f59741979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37476/0 (socket says 172.21.15.16:37476) 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 -- 172.21.15.16:0/322493451 learned_addr learned my addr 172.21.15.16:0/322493451 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5972d9d640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5974103360 0x7f597419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5971d9b640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59741029a0 0x7f5974197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 -- 172.21.15.16:0/322493451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59741029a0 msgr2=0x7f5974197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59741029a0 0x7f5974197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 -- 172.21.15.16:0/322493451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5974103360 msgr2=0x7f597419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5974103360 0x7f597419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 -- 172.21.15.16:0/322493451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597419f5d0 con 0x7f59741017b0 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5972d9d640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5974103360 0x7f597419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f5971d9b640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59741029a0 0x7f5974197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.794+0000 7f597259c640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f59741979b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59640314e0 tx=0x7f5964003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.795+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596403f070 con 0x7f59741017b0 2024-04-25T01:25:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.795+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597419f860 con 0x7f59741017b0 2024-04-25T01:25:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.795+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5964031c40 con 0x7f59741017b0 2024-04-25T01:25:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.795+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964035030 con 0x7f59741017b0 2024-04-25T01:25:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.795+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597419fda0 con 0x7f59741017b0 2024-04-25T01:25:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.796+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5974108ad0 con 0x7f59741017b0 2024-04-25T01:25:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.798+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5964042070 con 0x7f59741017b0 2024-04-25T01:25:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.798+0000 7f59637fe640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f595c0a2b90 0x7f595c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.798+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59640ef2b0 con 0x7f59741017b0 2024-04-25T01:25:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.798+0000 7f5971d9b640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f595c0a2b90 0x7f595c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.799+0000 7f5971d9b640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f595c0a2b90 0x7f595c0a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5974198ec0 tx=0x7f5968005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:12.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:12.803+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5964031df0 con 0x7f59741017b0 2024-04-25T01:25:12.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.006+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f597402d320 con 0x7f59741017b0 2024-04-25T01:25:12.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.007+0000 7f59637fe640 1 -- 172.21.15.16:0/322493451 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18433 (secure 0 0 0) 0x7f59640da820 con 0x7f59741017b0 2024-04-25T01:25:12.987 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:12.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:24:23.496690+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.009+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f595c0a2b90 msgr2=0x7f595c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.009+0000 7f5978802640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f595c0a2b90 0x7f595c0a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5974198ec0 tx=0x7f5968005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.009+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 msgr2=0x7f59741979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.009+0000 7f5978802640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f59741979b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59640314e0 tx=0x7f5964003c10 comp rx=0 tx=0).stop 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 shutdown_connections 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f595c0a2b90 0x7f595c0a5050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5974103360 0x7f597419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59741029a0 0x7f5974197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 --2- 172.21.15.16:0/322493451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59741017b0 0x7f59741979b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:12.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 >> 172.21.15.16:0/322493451 conn(0x7f59740fcd40 msgr2=0x7f5974105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:12.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 shutdown_connections 2024-04-25T01:25:12.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.010+0000 7f5978802640 1 -- 172.21.15.16:0/322493451 wait complete. 2024-04-25T01:25:13.001 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:25:13.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:25:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 -- 172.21.15.16:0/2000088484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30101870 msgr2=0x7fbe30101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 --2- 172.21.15.16:0/2000088484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30101870 0x7fbe30101cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009900 tx=0x7fbe2002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 -- 172.21.15.16:0/2000088484 shutdown_connections 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 --2- 172.21.15.16:0/2000088484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30102230 0x7fbe30108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 --2- 172.21.15.16:0/2000088484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30101870 0x7fbe30101cf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 --2- 172.21.15.16:0/2000088484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 0x7fbe30100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 -- 172.21.15.16:0/2000088484 >> 172.21.15.16:0/2000088484 conn(0x7fbe300fbbf0 msgr2=0x7fbe300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 -- 172.21.15.16:0/2000088484 shutdown_connections 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.127+0000 7fbe38038640 1 -- 172.21.15.16:0/2000088484 wait complete. 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 Processor -- start 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 -- start start 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 0x7fbe30104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30101870 0x7fbe30102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 0x7fbe30103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3010c190 con 0x7fbe30102230 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3010c010 con 0x7fbe30101870 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe38038640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3010c310 con 0x7fbe30100680 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe365d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 0x7fbe30103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe35dd4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 0x7fbe30104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe365d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 0x7fbe30103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37498/0 (socket says 172.21.15.16:37498) 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe365d5640 1 -- 172.21.15.16:0/3898417999 learned_addr learned my addr 172.21.15.16:0/3898417999 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.128+0000 7fbe355d3640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30101870 0x7fbe30102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe365d5640 1 -- 172.21.15.16:0/3898417999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 msgr2=0x7fbe30104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe365d5640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 0x7fbe30104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe365d5640 1 -- 172.21.15.16:0/3898417999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30101870 msgr2=0x7fbe30102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe365d5640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30101870 0x7fbe30102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe365d5640 1 -- 172.21.15.16:0/3898417999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe30103a20 con 0x7fbe30102230 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe35dd4640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 0x7fbe30104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe355d3640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30101870 0x7fbe30102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe365d5640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 0x7fbe30103190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800eb10 tx=0x7fbe2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28004b20 con 0x7fbe30102230 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe28004cc0 con 0x7fbe30102230 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe301a1b40 con 0x7fbe30102230 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28010690 con 0x7fbe30102230 2024-04-25T01:25:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.129+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe301a2060 con 0x7fbe30102230 2024-04-25T01:25:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.130+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe30068560 con 0x7fbe30102230 2024-04-25T01:25:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.131+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbe28005020 con 0x7fbe30102230 2024-04-25T01:25:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.132+0000 7fbe26ffd640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe1c0a2c10 0x7fbe1c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.132+0000 7fbe35dd4640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe1c0a2c10 0x7fbe1c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.132+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbe280cf1c0 con 0x7fbe30102230 2024-04-25T01:25:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.133+0000 7fbe35dd4640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe1c0a2c10 0x7fbe1c0a50d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe301016d0 tx=0x7fbe2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.137+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbe2809c520 con 0x7fbe30102230 2024-04-25T01:25:13.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:13.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/910854085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:13.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:13.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:13.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:12 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/910854085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:13.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:12 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.321+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbe301a2a20 con 0x7fbe30102230 2024-04-25T01:25:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.321+0000 7fbe26ffd640 1 -- 172.21.15.16:0/3898417999 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fbe2809be20 con 0x7fbe30102230 2024-04-25T01:25:13.302 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe1c0a2c10 msgr2=0x7fbe1c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe1c0a2c10 0x7fbe1c0a50d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe301016d0 tx=0x7fbe2c005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 msgr2=0x7fbe30103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 0x7fbe30103190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800eb10 tx=0x7fbe2800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 shutdown_connections 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe1c0a2c10 0x7fbe1c0a50d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe30102230 0x7fbe30103190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe30101870 0x7fbe30102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 --2- 172.21.15.16:0/3898417999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe30100680 0x7fbe30104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 >> 172.21.15.16:0/3898417999 conn(0x7fbe300fbbf0 msgr2=0x7fbe30068900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 shutdown_connections 2024-04-25T01:25:13.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.324+0000 7fbe38038640 1 -- 172.21.15.16:0/3898417999 wait complete. 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/910854085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:13.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:13.315 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:12 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:13.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:25:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 -- 172.21.15.16:0/2703868757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 msgr2=0x7f9ac4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 --2- 172.21.15.16:0/2703868757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac4108ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009ca0 tx=0x7f9abc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 -- 172.21.15.16:0/2703868757 shutdown_connections 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 --2- 172.21.15.16:0/2703868757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac4108ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 --2- 172.21.15.16:0/2703868757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac4101870 0x7f9ac4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 --2- 172.21.15.16:0/2703868757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ac4100680 0x7f9ac4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 -- 172.21.15.16:0/2703868757 >> 172.21.15.16:0/2703868757 conn(0x7f9ac40fbbf0 msgr2=0x7f9ac40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 -- 172.21.15.16:0/2703868757 shutdown_connections 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.438+0000 7f9acab14640 1 -- 172.21.15.16:0/2703868757 wait complete. 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 Processor -- start 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 -- start start 2024-04-25T01:25:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ac4100680 0x7f9ac4197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac4101870 0x7f9ac4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac410c190 con 0x7f9ac4102230 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac410c010 con 0x7f9ac4101870 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9acab14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac410c310 con 0x7f9ac4100680 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9ac90b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9ac3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac4101870 0x7f9ac4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9ac90b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37520/0 (socket says 172.21.15.16:37520) 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9ac90b1640 1 -- 172.21.15.16:0/2087330852 learned_addr learned my addr 172.21.15.16:0/2087330852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.439+0000 7f9ac88b0640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ac4100680 0x7f9ac4197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac90b1640 1 -- 172.21.15.16:0/2087330852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ac4100680 msgr2=0x7f9ac4197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac90b1640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ac4100680 0x7f9ac4197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac90b1640 1 -- 172.21.15.16:0/2087330852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac4101870 msgr2=0x7f9ac4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac90b1640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac4101870 0x7f9ac4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac90b1640 1 -- 172.21.15.16:0/2087330852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac419f560 con 0x7f9ac4102230 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac90b1640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac419eef0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009dc0 tx=0x7f9abc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc032ec0 con 0x7f9ac4102230 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9abc0044f0 con 0x7f9ac4102230 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac419f7f0 con 0x7f9ac4102230 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc030d40 con 0x7f9ac4102230 2024-04-25T01:25:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.440+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac419fd40 con 0x7f9ac4102230 2024-04-25T01:25:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.441+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac419ffa0 con 0x7f9ac4102230 2024-04-25T01:25:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.442+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9abc004040 con 0x7f9ac4102230 2024-04-25T01:25:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.443+0000 7f9ac1ffb640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ab40a2d30 0x7f9ab40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.443+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9abc0f02d0 con 0x7f9ac4102230 2024-04-25T01:25:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.443+0000 7f9ac88b0640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ab40a2d30 0x7f9ab40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.444+0000 7f9ac88b0640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ab40a2d30 0x7f9ab40a51f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9ac41016d0 tx=0x7f9ab800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.447+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9abc0bd6b0 con 0x7f9ac4102230 2024-04-25T01:25:13.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.638+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9ac41a0330 con 0x7f9ac4102230 2024-04-25T01:25:13.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.639+0000 7f9ac1ffb640 1 -- 172.21.15.16:0/2087330852 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f9abc0bcfb0 con 0x7f9ac4102230 2024-04-25T01:25:13.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:13.619 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-04-25T01:25:13.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:25:13.619 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-04-25T01:25:13.620 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-04-25T01:25:13.620 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:25:13.620 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ab40a2d30 msgr2=0x7f9ab40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ab40a2d30 0x7f9ab40a51f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9ac41016d0 tx=0x7f9ab800b390 comp rx=0 tx=0).stop 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 msgr2=0x7f9ac419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac419eef0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009dc0 tx=0x7f9abc003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 shutdown_connections 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ab40a2d30 0x7f9ab40a51f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ac4102230 0x7f9ac419eef0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac4101870 0x7f9ac4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 --2- 172.21.15.16:0/2087330852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ac4100680 0x7f9ac4197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 >> 172.21.15.16:0/2087330852 conn(0x7f9ac40fbbf0 msgr2=0x7f9ac40fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 shutdown_connections 2024-04-25T01:25:13.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.642+0000 7f9acab14640 1 -- 172.21.15.16:0/2087330852 wait complete. 2024-04-25T01:25:13.634 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:25:13.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:25:13.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 -- 172.21.15.16:0/31678897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 msgr2=0x7f8be4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 --2- 172.21.15.16:0/31678897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4101cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009900 tx=0x7f8bd402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 -- 172.21.15.16:0/31678897 shutdown_connections 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 --2- 172.21.15.16:0/31678897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8be4102230 0x7f8be4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 --2- 172.21.15.16:0/31678897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4101cf0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 --2- 172.21.15.16:0/31678897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8be4100680 0x7f8be4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 -- 172.21.15.16:0/31678897 >> 172.21.15.16:0/31678897 conn(0x7f8be40fbbf0 msgr2=0x7f8be40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 -- 172.21.15.16:0/31678897 shutdown_connections 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.766+0000 7f8bebd09640 1 -- 172.21.15.16:0/31678897 wait complete. 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 Processor -- start 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 -- start start 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8be4100680 0x7f8be4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8be4102230 0x7f8be419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8be410c1b0 con 0x7f8be4101870 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8be410c030 con 0x7f8be4102230 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bebd09640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8be410c330 con 0x7f8be4100680 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37536/0 (socket says 172.21.15.16:37536) 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 -- 172.21.15.16:0/1279886865 learned_addr learned my addr 172.21.15.16:0/1279886865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be9aa5640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8be4100680 0x7f8be4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8bea2a6640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8be4102230 0x7f8be419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 -- 172.21.15.16:0/1279886865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8be4100680 msgr2=0x7f8be4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8be4100680 0x7f8be4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 -- 172.21.15.16:0/1279886865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8be4102230 msgr2=0x7f8be419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8be4102230 0x7f8be419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.767+0000 7f8be92a4640 1 -- 172.21.15.16:0/1279886865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be419f660 con 0x7f8be4101870 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.768+0000 7f8be92a4640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4197fd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4004770 tx=0x7f8bd40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.768+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd403f030 con 0x7f8be4101870 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.768+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bd4003cb0 con 0x7f8be4101870 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.768+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be419f8f0 con 0x7f8be4101870 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.768+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4035070 con 0x7f8be4101870 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.768+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be419fe30 con 0x7f8be4101870 2024-04-25T01:25:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.769+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be41a0060 con 0x7f8be4101870 2024-04-25T01:25:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.770+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8bd4042030 con 0x7f8be4101870 2024-04-25T01:25:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.771+0000 7f8bdaffd640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8bd00a2c10 0x7f8bd00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.771+0000 7f8be9aa5640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8bd00a2c10 0x7f8bd00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.771+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8bd40ef4f0 con 0x7f8be4101870 2024-04-25T01:25:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.771+0000 7f8be9aa5640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8bd00a2c10 0x7f8bd00a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8be0005fd0 tx=0x7f8be000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.775+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8bd40bc200 con 0x7f8be4101870 2024-04-25T01:25:13.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:13.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8be41a0230 con 0x7f8be4101870 2024-04-25T01:25:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:14 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/322493451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:14 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:14 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3898417999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:14 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2087330852' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:25:14.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:14 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/322493451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:14 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:14 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3898417999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:14 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2087330852' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:25:14.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:14 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/322493451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:14 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:14.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:14 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3898417999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:25:14.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:14 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2087330852' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:25:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.959+0000 7f8bdaffd640 1 -- 172.21.15.16:0/1279886865 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f8bd40c0500 con 0x7f8be4101870 2024-04-25T01:25:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-04-25T01:25:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8bd00a2c10 msgr2=0x7f8bd00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8bd00a2c10 0x7f8bd00a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8be0005fd0 tx=0x7f8be000b390 comp rx=0 tx=0).stop 2024-04-25T01:25:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 msgr2=0x7f8be4197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4197fd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4004770 tx=0x7f8bd40042f0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 shutdown_connections 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8bd00a2c10 0x7f8bd00a50d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8be4102230 0x7f8be419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8be4101870 0x7f8be4197fd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 --2- 172.21.15.16:0/1279886865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8be4100680 0x7f8be4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 >> 172.21.15.16:0/1279886865 conn(0x7f8be40fbbf0 msgr2=0x7f8be40688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 shutdown_connections 2024-04-25T01:25:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:14.961+0000 7f8bebd09640 1 -- 172.21.15.16:0/1279886865 wait complete. 2024-04-25T01:25:14.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:25:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 -- 172.21.15.16:0/330059381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc102200 msgr2=0x7f8ffc1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 --2- 172.21.15.16:0/330059381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc102200 0x7f8ffc1045f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009900 tx=0x7f8ff402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 -- 172.21.15.16:0/330059381 shutdown_connections 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 --2- 172.21.15.16:0/330059381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc107460 0x7f8ffc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 --2- 172.21.15.16:0/330059381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ffc104b30 0x7f8ffc106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 --2- 172.21.15.16:0/330059381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc102200 0x7f8ffc1045f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.088+0000 7f9000bc1640 1 -- 172.21.15.16:0/330059381 >> 172.21.15.16:0/330059381 conn(0x7f8ffc0fbbf0 msgr2=0x7f8ffc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 -- 172.21.15.16:0/330059381 shutdown_connections 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 -- 172.21.15.16:0/330059381 wait complete. 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 Processor -- start 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 -- start start 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ffc102200 0x7f8ffc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc104b30 0x7f8ffc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 0x7f8ffc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc10d320 con 0x7f8ffc107460 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc10d1a0 con 0x7f8ffc104b30 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.089+0000 7f9000bc1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc10d4a0 con 0x7f8ffc102200 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 0x7f8ffc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 0x7f8ffc19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37554/0 (socket says 172.21.15.16:37554) 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 -- 172.21.15.16:0/3008060212 learned_addr learned my addr 172.21.15.16:0/3008060212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ff9d9b640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc104b30 0x7f8ffc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffa59c640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ffc102200 0x7f8ffc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 -- 172.21.15.16:0/3008060212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ffc102200 msgr2=0x7f8ffc197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ffc102200 0x7f8ffc197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 -- 172.21.15.16:0/3008060212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc104b30 msgr2=0x7f8ffc198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc104b30 0x7f8ffc198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 -- 172.21.15.16:0/3008060212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc19f690 con 0x7f8ffc107460 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ffad9d640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 0x7f8ffc19f020 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00b970 tx=0x7f8fec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec019070 con 0x7f8ffc107460 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fec014440 con 0x7f8ffc107460 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ffc19f980 con 0x7f8ffc107460 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0075d0 con 0x7f8ffc107460 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.090+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ffc19ff50 con 0x7f8ffc107460 2024-04-25T01:25:15.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.091+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ffc0fd6e0 con 0x7f8ffc107460 2024-04-25T01:25:15.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.093+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8fec01c070 con 0x7f8ffc107460 2024-04-25T01:25:15.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.093+0000 7f8ff37fe640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8fe80a2c10 0x7f8fe80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:15.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.093+0000 7f8ffa59c640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8fe80a2c10 0x7f8fe80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:15.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.093+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f8fec0ce6c0 con 0x7f8ffc107460 2024-04-25T01:25:15.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.094+0000 7f8ffa59c640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8fe80a2c10 0x7f8fe80a50d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4000c00 tx=0x7f8ff40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:15.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.097+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8fec09b300 con 0x7f8ffc107460 2024-04-25T01:25:15.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.287+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8ffc02d170 con 0x7f8ffc107460 2024-04-25T01:25:15.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.961+0000 7f8ff37fe640 1 -- 172.21.15.16:0/3008060212 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f8fec09f600 con 0x7f8ffc107460 2024-04-25T01:25:15.941 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-04-25T01:25:15.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8fe80a2c10 msgr2=0x7f8fe80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8fe80a2c10 0x7f8fe80a50d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4000c00 tx=0x7f8ff40314a0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 msgr2=0x7f8ffc19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:15.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 0x7f8ffc19f020 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00b970 tx=0x7f8fec00be30 comp rx=0 tx=0).stop 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 shutdown_connections 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8fe80a2c10 0x7f8fe80a50d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ffc107460 0x7f8ffc19f020 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc104b30 0x7f8ffc198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 --2- 172.21.15.16:0/3008060212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ffc102200 0x7f8ffc197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 >> 172.21.15.16:0/3008060212 conn(0x7f8ffc0fbbf0 msgr2=0x7f8ffc102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 shutdown_connections 2024-04-25T01:25:15.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:15.963+0000 7f9000bc1640 1 -- 172.21.15.16:0/3008060212 wait complete. 2024-04-25T01:25:15.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1279886865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1279886865' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:15.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:15 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:25:15.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:15.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1279886865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:25:15.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1279886865' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:15.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:15 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1279886865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1279886865' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:16.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:16.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:15 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:25:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 -- 172.21.15.16:0/2461987946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130104b50 msgr2=0x7f8130106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 --2- 172.21.15.16:0/2461987946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130104b50 0x7f8130106f40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8118009900 tx=0x7f811802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 -- 172.21.15.16:0/2461987946 shutdown_connections 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 --2- 172.21.15.16:0/2461987946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130107480 0x7f81301098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 --2- 172.21.15.16:0/2461987946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130104b50 0x7f8130106f40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 --2- 172.21.15.16:0/2461987946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130102220 0x7f8130104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 -- 172.21.15.16:0/2461987946 >> 172.21.15.16:0/2461987946 conn(0x7f81300fbc30 msgr2=0x7f81300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 -- 172.21.15.16:0/2461987946 shutdown_connections 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.092+0000 7f8136b70640 1 -- 172.21.15.16:0/2461987946 wait complete. 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 Processor -- start 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 -- start start 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 0x7f8130197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130104b50 0x7f8130197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130107480 0x7f813019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f813010d300 con 0x7f8130102220 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f813010d180 con 0x7f8130107480 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f8136b70640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f813010d480 con 0x7f8130104b50 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f813490c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 0x7f8130197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f813490c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 0x7f8130197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37570/0 (socket says 172.21.15.16:37570) 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f813490c640 1 -- 172.21.15.16:0/2480766200 learned_addr learned my addr 172.21.15.16:0/2480766200 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f813510d640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130107480 0x7f813019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.093+0000 7f812ffff640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130104b50 0x7f8130197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813490c640 1 -- 172.21.15.16:0/2480766200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130104b50 msgr2=0x7f8130197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813490c640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130104b50 0x7f8130197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813490c640 1 -- 172.21.15.16:0/2480766200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130107480 msgr2=0x7f813019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813490c640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130107480 0x7f813019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813490c640 1 -- 172.21.15.16:0/2480766200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813019f650 con 0x7f8130102220 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f812ffff640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130104b50 0x7f8130197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813510d640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130107480 0x7f813019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f813490c640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 0x7f8130197a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f812000e9f0 tx=0x7f812000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812000cd80 con 0x7f8130102220 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81200108e0 con 0x7f8130102220 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813019f940 con 0x7f8130102220 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120018920 con 0x7f8130102220 2024-04-25T01:25:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.094+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f813019fee0 con 0x7f8130102220 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.095+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81300fd720 con 0x7f8130102220 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.096+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8120010a90 con 0x7f8130102220 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.097+0000 7f812dffb640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f811c0a2c10 0x7f811c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.097+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f81200cf6e0 con 0x7f8130102220 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.097+0000 7f812ffff640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f811c0a2c10 0x7f811c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.098+0000 7f812ffff640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f811c0a2c10 0x7f811c0a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f811802a930 tx=0x7f8118005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.101+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f812009c7f0 con 0x7f8130102220 2024-04-25T01:25:16.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:16.326+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f813002d170 con 0x7f8130102220 2024-04-25T01:25:16.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3008060212' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:25:16.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: pgmap v290: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:16.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3008060212' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:25:16.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:25:16.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:25:16.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2480766200' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:25:16.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:16 smithi038 ceph-mon[32841]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3008060212' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: pgmap v290: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3008060212' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2480766200' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:25:17.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:16 smithi156 ceph-mon[32631]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:25:17.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3008060212' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:25:17.002 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: pgmap v290: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:17.003 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3008060212' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:25:17.003 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:25:17.003 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:25:17.003 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2480766200' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:25:17.003 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:16 smithi016 ceph-mon[29153]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:25:17.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.038+0000 7f812dffb640 1 -- 172.21.15.16:0/2480766200 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f812009c0f0 con 0x7f8130102220 2024-04-25T01:25:17.018 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:25:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f811c0a2c10 msgr2=0x7f811c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f811c0a2c10 0x7f811c0a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f811802a930 tx=0x7f8118005fd0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 msgr2=0x7f8130197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 0x7f8130197a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f812000e9f0 tx=0x7f812000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 shutdown_connections 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f811c0a2c10 0x7f811c0a50d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8130107480 0x7f813019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8130104b50 0x7f8130197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 --2- 172.21.15.16:0/2480766200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8130102220 0x7f8130197a30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 >> 172.21.15.16:0/2480766200 conn(0x7f81300fbc30 msgr2=0x7f8130102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.041+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 shutdown_connections 2024-04-25T01:25:17.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.042+0000 7f8136b70640 1 -- 172.21.15.16:0/2480766200 wait complete. 2024-04-25T01:25:17.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 -- 172.21.15.16:0/3512482215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 msgr2=0x7fd2d0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/3512482215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0100aa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009900 tx=0x7fd2c402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 -- 172.21.15.16:0/3512482215 shutdown_connections 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/3512482215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0102230 0x7fd2d0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/3512482215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2d0101870 0x7fd2d0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/3512482215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0100aa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 -- 172.21.15.16:0/3512482215 >> 172.21.15.16:0/3512482215 conn(0x7fd2d00fbbf0 msgr2=0x7fd2d00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 -- 172.21.15.16:0/3512482215 shutdown_connections 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.159+0000 7fd2d6f77640 1 -- 172.21.15.16:0/3512482215 wait complete. 2024-04-25T01:25:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 Processor -- start 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 -- start start 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0101870 0x7fd2d019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2d0102230 0x7fd2d01a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d010d310 con 0x7fd2d0100680 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d010d190 con 0x7fd2d0101870 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d6f77640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d010d490 con 0x7fd2d0102230 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d4d13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d4d13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37596/0 (socket says 172.21.15.16:37596) 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2d4d13640 1 -- 172.21.15.16:0/1339491410 learned_addr learned my addr 172.21.15.16:0/1339491410 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.160+0000 7fd2cffff640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0101870 0x7fd2d019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d5514640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2d0102230 0x7fd2d01a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d4d13640 1 -- 172.21.15.16:0/1339491410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2d0102230 msgr2=0x7fd2d01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d4d13640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2d0102230 0x7fd2d01a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d4d13640 1 -- 172.21.15.16:0/1339491410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0101870 msgr2=0x7fd2d019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d4d13640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0101870 0x7fd2d019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d4d13640 1 -- 172.21.15.16:0/1339491410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d01a18b0 con 0x7fd2d0100680 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2cffff640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0101870 0x7fd2d019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d4d13640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0199c70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40314e0 tx=0x7fd2c40042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c403f030 con 0x7fd2d0100680 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c4003cb0 con 0x7fd2d0100680 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d01a1b40 con 0x7fd2d0100680 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4035070 con 0x7fd2d0100680 2024-04-25T01:25:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.161+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d01a2080 con 0x7fd2d0100680 2024-04-25T01:25:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.162+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d01a22b0 con 0x7fd2d0100680 2024-04-25T01:25:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.163+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd2c4042030 con 0x7fd2d0100680 2024-04-25T01:25:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.164+0000 7fd2cdffb640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd2c00a2c10 0x7fd2c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.164+0000 7fd2cffff640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd2c00a2c10 0x7fd2c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.164+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fd2c40ef600 con 0x7fd2d0100680 2024-04-25T01:25:17.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.165+0000 7fd2cffff640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd2c00a2c10 0x7fd2c00a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd2d019b170 tx=0x7fd2bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.169+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd2c40bc610 con 0x7fd2d0100680 2024-04-25T01:25:17.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.353+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2d01a2640 con 0x7fd2d0100680 2024-04-25T01:25:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.355+0000 7fd2cdffb640 1 -- 172.21.15.16:0/1339491410 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22209 (secure 0 0 0) 0x7fd2c40bbf10 con 0x7fd2d0100680 2024-04-25T01:25:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:17.335 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:25:16.327593+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd2c00a2c10 msgr2=0x7fd2c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd2c00a2c10 0x7fd2c00a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd2d019b170 tx=0x7fd2bc008040 comp rx=0 tx=0).stop 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 msgr2=0x7fd2d0199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0199c70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40314e0 tx=0x7fd2c40042e0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 shutdown_connections 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd2c00a2c10 0x7fd2c00a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2d0102230 0x7fd2d01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2d0101870 0x7fd2d019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 --2- 172.21.15.16:0/1339491410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d0100680 0x7fd2d0199c70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 >> 172.21.15.16:0/1339491410 conn(0x7fd2d00fbbf0 msgr2=0x7fd2d0107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 shutdown_connections 2024-04-25T01:25:17.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.357+0000 7fd2d6f77640 1 -- 172.21.15.16:0/1339491410 wait complete. 2024-04-25T01:25:17.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.469+0000 7f9ac4363640 1 -- 172.21.15.16:0/3195601399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc101870 msgr2=0x7f9abc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.469+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3195601399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc101870 0x7f9abc101cf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009900 tx=0x7f9ab802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- 172.21.15.16:0/3195601399 shutdown_connections 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3195601399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc102230 0x7f9abc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3195601399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc101870 0x7f9abc101cf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3195601399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc100680 0x7f9abc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- 172.21.15.16:0/3195601399 >> 172.21.15.16:0/3195601399 conn(0x7f9abc0fbbf0 msgr2=0x7f9abc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- 172.21.15.16:0/3195601399 shutdown_connections 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- 172.21.15.16:0/3195601399 wait complete. 2024-04-25T01:25:17.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 Processor -- start 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- start start 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc100680 0x7f9abc197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc101870 0x7f9abc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 0x7f9abc19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc10c1b0 con 0x7f9abc102230 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc10c030 con 0x7f9abc101870 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac4363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc10c330 con 0x7f9abc100680 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.470+0000 7f9ac2900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 0x7f9abc19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 0x7f9abc19efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37622/0 (socket says 172.21.15.16:37622) 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 -- 172.21.15.16:0/3550759756 learned_addr learned my addr 172.21.15.16:0/3550759756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac18fe640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc101870 0x7f9abc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac20ff640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc100680 0x7f9abc197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 -- 172.21.15.16:0/3550759756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc100680 msgr2=0x7f9abc197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc100680 0x7f9abc197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 -- 172.21.15.16:0/3550759756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc101870 msgr2=0x7f9abc197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc101870 0x7f9abc197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 -- 172.21.15.16:0/3550759756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc19f620 con 0x7f9abc102230 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac20ff640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc100680 0x7f9abc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac2900640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 0x7f9abc19efb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400cad0 tx=0x7f9ab400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab400ae20 con 0x7f9abc102230 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ab4005020 con 0x7f9abc102230 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9abc19f910 con 0x7f9abc102230 2024-04-25T01:25:17.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab400f6b0 con 0x7f9abc102230 2024-04-25T01:25:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.471+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9abc19fe80 con 0x7f9abc102230 2024-04-25T01:25:17.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.472+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9abc1a00b0 con 0x7f9abc102230 2024-04-25T01:25:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.473+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9ab40051d0 con 0x7f9abc102230 2024-04-25T01:25:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.474+0000 7f9ab37fe640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9aa80a2c10 0x7f9aa80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.474+0000 7f9ac20ff640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9aa80a2c10 0x7f9aa80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.474+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f9ab40cef60 con 0x7f9abc102230 2024-04-25T01:25:17.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.474+0000 7f9ac20ff640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9aa80a2c10 0x7f9aa80a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0058c0 tx=0x7f9aac005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.478+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9ab4014030 con 0x7f9abc102230 2024-04-25T01:25:17.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.666+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9abc1a0320 con 0x7f9abc102230 2024-04-25T01:25:17.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.667+0000 7f9ab37fe640 1 -- 172.21.15.16:0/3550759756 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22209 (secure 0 0 0) 0x7f9ab409bf70 con 0x7f9abc102230 2024-04-25T01:25:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:17.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:25:16.327593+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.669+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9aa80a2c10 msgr2=0x7f9aa80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.669+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9aa80a2c10 0x7f9aa80a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0058c0 tx=0x7f9aac005850 comp rx=0 tx=0).stop 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.669+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 msgr2=0x7f9abc19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.669+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 0x7f9abc19efb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400cad0 tx=0x7f9ab400cf90 comp rx=0 tx=0).stop 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.669+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 shutdown_connections 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.669+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9aa80a2c10 0x7f9aa80a50d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.670+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9abc102230 0x7f9abc19efb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.670+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc101870 0x7f9abc197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.670+0000 7f9ac4363640 1 --2- 172.21.15.16:0/3550759756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9abc100680 0x7f9abc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.670+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 >> 172.21.15.16:0/3550759756 conn(0x7f9abc0fbbf0 msgr2=0x7f9abc068900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.670+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 shutdown_connections 2024-04-25T01:25:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.670+0000 7f9ac4363640 1 -- 172.21.15.16:0/3550759756 wait complete. 2024-04-25T01:25:17.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 -- 172.21.15.16:0/145024568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 msgr2=0x7f0448100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 --2- 172.21.15.16:0/145024568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448100aa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0438009900 tx=0x7f043802a410 comp rx=0 tx=0).stop 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 -- 172.21.15.16:0/145024568 shutdown_connections 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 --2- 172.21.15.16:0/145024568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 0x7f0448108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 --2- 172.21.15.16:0/145024568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0448101870 0x7f0448101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 --2- 172.21.15.16:0/145024568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448100aa0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 -- 172.21.15.16:0/145024568 >> 172.21.15.16:0/145024568 conn(0x7f04480fbbf0 msgr2=0x7f04480fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 -- 172.21.15.16:0/145024568 shutdown_connections 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.788+0000 7f044d63f640 1 -- 172.21.15.16:0/145024568 wait complete. 2024-04-25T01:25:17.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 Processor -- start 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 -- start start 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0448101870 0x7f044819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 0x7f04481a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f044810d2d0 con 0x7f0448100680 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f044810d150 con 0x7f0448101870 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f044d63f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f044810d450 con 0x7f0448102230 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f0446ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f0446ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37636/0 (socket says 172.21.15.16:37636) 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f0446ffd640 1 -- 172.21.15.16:0/1447898299 learned_addr learned my addr 172.21.15.16:0/1447898299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.789+0000 7f043ffff640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0448101870 0x7f044819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f04477fe640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 0x7f04481a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0446ffd640 1 -- 172.21.15.16:0/1447898299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 msgr2=0x7f04481a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0446ffd640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 0x7f04481a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0446ffd640 1 -- 172.21.15.16:0/1447898299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0448101870 msgr2=0x7f044819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0446ffd640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0448101870 0x7f044819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0446ffd640 1 -- 172.21.15.16:0/1447898299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04481a1840 con 0x7f0448100680 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f04477fe640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 0x7f04481a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0446ffd640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448199c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f04380314e0 tx=0x7f04380042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043803f030 con 0x7f0448100680 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04481a1ad0 con 0x7f0448100680 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0438003cb0 con 0x7f0448100680 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438035070 con 0x7f0448100680 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.790+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04481a2010 con 0x7f0448100680 2024-04-25T01:25:17.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.791+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04481a21e0 con 0x7f0448100680 2024-04-25T01:25:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.792+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0438042030 con 0x7f0448100680 2024-04-25T01:25:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.793+0000 7f0444ff9640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f04340a2b90 0x7f04340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:17.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.793+0000 7f043ffff640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f04340a2b90 0x7f04340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.793+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f04380ef9f0 con 0x7f0448100680 2024-04-25T01:25:17.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.793+0000 7f043ffff640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f04340a2b90 0x7f04340a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f043000bbe0 tx=0x7f043000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:17.796+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f04380bcaa0 con 0x7f0448100680 2024-04-25T01:25:17.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:17 smithi038 ceph-mon[32841]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:25:17.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2480766200' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:25:17.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:17 smithi038 ceph-mon[32841]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:25:17.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:17 smithi038 ceph-mon[32841]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:25:17.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1339491410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:17.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3550759756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:18.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:17 smithi156 ceph-mon[32631]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:25:18.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2480766200' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:25:18.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:17 smithi156 ceph-mon[32631]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:25:18.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:17 smithi156 ceph-mon[32631]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:25:18.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1339491410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:18.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3550759756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:18.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.025+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f04481a23b0 con 0x7f0448100680 2024-04-25T01:25:18.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.048+0000 7f0444ff9640 1 -- 172.21.15.16:0/1447898299 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f04380bc3a0 con 0x7f0448100680 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f04340a2b90 msgr2=0x7f04340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f04340a2b90 0x7f04340a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f043000bbe0 tx=0x7f043000b390 comp rx=0 tx=0).stop 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 msgr2=0x7f0448199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448199c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f04380314e0 tx=0x7f04380042e0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 shutdown_connections 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f04340a2b90 0x7f04340a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0448102230 0x7f04481a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0448101870 0x7f044819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 --2- 172.21.15.16:0/1447898299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0448100680 0x7f0448199c30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 >> 172.21.15.16:0/1447898299 conn(0x7f04480fbbf0 msgr2=0x7f0448109300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 shutdown_connections 2024-04-25T01:25:18.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.051+0000 7f044d63f640 1 -- 172.21.15.16:0/1447898299 wait complete. 2024-04-25T01:25:18.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:17 smithi016 ceph-mon[29153]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:25:18.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2480766200' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:25:18.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:17 smithi016 ceph-mon[29153]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:25:18.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:17 smithi016 ceph-mon[29153]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:25:18.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1339491410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:18.041 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3550759756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:18.042 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-25T01:25:18.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-25T01:25:18.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 -- 172.21.15.16:0/3013849226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc0102990 msgr2=0x7f1bc0102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 --2- 172.21.15.16:0/3013849226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc0102990 0x7f1bc0102e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009900 tx=0x7f1bb002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:18.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 -- 172.21.15.16:0/3013849226 shutdown_connections 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 --2- 172.21.15.16:0/3013849226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 0x7f1bc0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 --2- 172.21.15.16:0/3013849226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc0102990 0x7f1bc0102e10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 --2- 172.21.15.16:0/3013849226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bc01017a0 0x7f1bc0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 -- 172.21.15.16:0/3013849226 >> 172.21.15.16:0/3013849226 conn(0x7f1bc00fcd10 msgr2=0x7f1bc00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 -- 172.21.15.16:0/3013849226 shutdown_connections 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 -- 172.21.15.16:0/3013849226 wait complete. 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 Processor -- start 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.167+0000 7f1bc7295640 1 -- start start 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 0x7f1bc00754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bc0102990 0x7f1bc0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 0x7f1bc0074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010d2d0 con 0x7f1bc01017a0 2024-04-25T01:25:18.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010d150 con 0x7f1bc0103350 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010d450 con 0x7f1bc0102990 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 0x7f1bc00754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 0x7f1bc00754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37660/0 (socket says 172.21.15.16:37660) 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 -- 172.21.15.16:0/1545367040 learned_addr learned my addr 172.21.15.16:0/1545367040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5832640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 0x7f1bc0074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc4830640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bc0102990 0x7f1bc0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 -- 172.21.15.16:0/1545367040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bc0102990 msgr2=0x7f1bc0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bc0102990 0x7f1bc0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 -- 172.21.15.16:0/1545367040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 msgr2=0x7f1bc0074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 0x7f1bc0074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 -- 172.21.15.16:0/1545367040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc0074900 con 0x7f1bc01017a0 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5832640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 0x7f1bc0074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc5031640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 0x7f1bc00754f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00d950 tx=0x7f1bbc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc017070 con 0x7f1bc01017a0 2024-04-25T01:25:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bbc009c30 con 0x7f1bc01017a0 2024-04-25T01:25:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc01a1af0 con 0x7f1bc01017a0 2024-04-25T01:25:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00f620 con 0x7f1bc01017a0 2024-04-25T01:25:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.168+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc01a2040 con 0x7f1bc01017a0 2024-04-25T01:25:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.169+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc0108b60 con 0x7f1bc01017a0 2024-04-25T01:25:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.170+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1bbc01a070 con 0x7f1bc01017a0 2024-04-25T01:25:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.171+0000 7f1bb67fc640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bac0a2c10 0x7f1bac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.171+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1bbc0ce990 con 0x7f1bc01017a0 2024-04-25T01:25:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.171+0000 7f1bc4830640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bac0a2c10 0x7f1bac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.172+0000 7f1bc4830640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bac0a2c10 0x7f1bac0a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0075260 tx=0x7f1bb00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:18.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.176+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1bbc009de0 con 0x7f1bc01017a0 2024-04-25T01:25:18.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:18.400+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f1bc002d170 con 0x7f1bc01017a0 2024-04-25T01:25:18.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: pgmap v293: 97 pgs: 20 active+clean, 21 creating+peering, 56 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:18.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:25:18.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1447898299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:25:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:25:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: Cluster is now healthy 2024-04-25T01:25:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1447898299' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:25:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] up:active 2024-04-25T01:25:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:25:18.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:18 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1545367040' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: pgmap v293: 97 pgs: 20 active+clean, 21 creating+peering, 56 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1447898299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: Cluster is now healthy 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1447898299' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] up:active 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:25:19.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:18 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1545367040' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:25:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.050+0000 7f1bb67fc640 1 -- 172.21.15.16:0/1545367040 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f1bbc09ba40 con 0x7f1bc01017a0 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bac0a2c10 msgr2=0x7f1bac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bac0a2c10 0x7f1bac0a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0075260 tx=0x7f1bb00314a0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 msgr2=0x7f1bc00754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 0x7f1bc00754f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00d950 tx=0x7f1bbc00de10 comp rx=0 tx=0).stop 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 shutdown_connections 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bac0a2c10 0x7f1bac0a50d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0103350 0x7f1bc0074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bc0102990 0x7f1bc0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 --2- 172.21.15.16:0/1545367040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bc01017a0 0x7f1bc00754f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 >> 172.21.15.16:0/1545367040 conn(0x7f1bc00fcd10 msgr2=0x7f1bc0105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:19.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 shutdown_connections 2024-04-25T01:25:19.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.053+0000 7f1bc7295640 1 -- 172.21.15.16:0/1545367040 wait complete. 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: pgmap v293: 97 pgs: 20 active+clean, 21 creating+peering, 56 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1447898299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: Cluster is now healthy 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1447898299' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] up:active 2024-04-25T01:25:19.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:25:19.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:18 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1545367040' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:25:19.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 -- 172.21.15.16:0/1970409003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e81006a0 msgr2=0x7f09e8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 --2- 172.21.15.16:0/1970409003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e81006a0 0x7f09e8100ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009900 tx=0x7f09e402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 -- 172.21.15.16:0/1970409003 shutdown_connections 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 --2- 172.21.15.16:0/1970409003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09e8102250 0x7f09e8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 --2- 172.21.15.16:0/1970409003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8101890 0x7f09e8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 --2- 172.21.15.16:0/1970409003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e81006a0 0x7f09e8100ac0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 -- 172.21.15.16:0/1970409003 >> 172.21.15.16:0/1970409003 conn(0x7f09e80fbc10 msgr2=0x7f09e80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 -- 172.21.15.16:0/1970409003 shutdown_connections 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 -- 172.21.15.16:0/1970409003 wait complete. 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 Processor -- start 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 -- start start 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.182+0000 7f09f035a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09e81006a0 0x7f09e8197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 0x7f09e81980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8102250 0x7f09e819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09e810c2a0 con 0x7f09e8101890 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09e810c120 con 0x7f09e8102250 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09e810c420 con 0x7f09e81006a0 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 0x7f09e81980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 0x7f09e81980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37668/0 (socket says 172.21.15.16:37668) 2024-04-25T01:25:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 -- 172.21.15.16:0/74823016 learned_addr learned my addr 172.21.15.16:0/74823016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ee8f7640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8102250 0x7f09e819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ee0f6640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09e81006a0 0x7f09e8197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 -- 172.21.15.16:0/74823016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09e81006a0 msgr2=0x7f09e8197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09e81006a0 0x7f09e8197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 -- 172.21.15.16:0/74823016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8102250 msgr2=0x7f09e819f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8102250 0x7f09e819f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 -- 172.21.15.16:0/74823016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e819f7a0 con 0x7f09e8101890 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ee8f7640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8102250 0x7f09e819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09ed8f5640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 0x7f09e81980c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09d800b4a0 tx=0x7f09d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8014070 con 0x7f09e8101890 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09d8010ac0 con 0x7f09e8101890 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e819fa90 con 0x7f09e8101890 2024-04-25T01:25:19.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8017c30 con 0x7f09e8101890 2024-04-25T01:25:19.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.183+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e81a0060 con 0x7f09e8101890 2024-04-25T01:25:19.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.184+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e81a02f0 con 0x7f09e8101890 2024-04-25T01:25:19.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.186+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f09d8017070 con 0x7f09e8101890 2024-04-25T01:25:19.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.186+0000 7f09df7fe640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09d40a2b90 0x7f09d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:19.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.186+0000 7f09ee0f6640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09d40a2b90 0x7f09d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:19.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.186+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f09d80ce750 con 0x7f09e8101890 2024-04-25T01:25:19.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.187+0000 7f09ee0f6640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09d40a2b90 0x7f09d40a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f09e4000c00 tx=0x7f09e40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:19.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.190+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f09d809b760 con 0x7f09e8101890 2024-04-25T01:25:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:19.417+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f09e81a06e0 con 0x7f09e8101890 2024-04-25T01:25:19.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:25:19.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1545367040' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:25:19.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:25:19.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:25:19.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:25:19.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:25:19.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:19 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/74823016' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1545367040' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:25:20.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:19 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/74823016' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:25:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.056+0000 7f09df7fe640 1 -- 172.21.15.16:0/74823016 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f09d809b060 con 0x7f09e8101890 2024-04-25T01:25:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09d40a2b90 msgr2=0x7f09d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09d40a2b90 0x7f09d40a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f09e4000c00 tx=0x7f09e40314a0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 msgr2=0x7f09e81980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 0x7f09e81980c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09d800b4a0 tx=0x7f09d800b960 comp rx=0 tx=0).stop 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 shutdown_connections 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09d40a2b90 0x7f09d40a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09e8102250 0x7f09e819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09e8101890 0x7f09e81980c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 --2- 172.21.15.16:0/74823016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09e81006a0 0x7f09e8197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.059+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 >> 172.21.15.16:0/74823016 conn(0x7f09e80fbc10 msgr2=0x7f09e8069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.060+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 shutdown_connections 2024-04-25T01:25:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.060+0000 7f09f035a640 1 -- 172.21.15.16:0/74823016 wait complete. 2024-04-25T01:25:20.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:25:20.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1545367040' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:25:20.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:25:20.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:25:20.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:25:20.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:25:20.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:19 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/74823016' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:25:20.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.187+0000 7f0bdaffb640 1 -- 172.21.15.16:0/2969211357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4107460 msgr2=0x7f0bd41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.187+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/2969211357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4107460 0x7f0bd41098b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009ca0 tx=0x7f0bcc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- 172.21.15.16:0/2969211357 shutdown_connections 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/2969211357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4107460 0x7f0bd41098b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/2969211357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bd4104b30 0x7f0bd4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/2969211357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bd4102200 0x7f0bd41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- 172.21.15.16:0/2969211357 >> 172.21.15.16:0/2969211357 conn(0x7f0bd40fbbf0 msgr2=0x7f0bd40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:20.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- 172.21.15.16:0/2969211357 shutdown_connections 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- 172.21.15.16:0/2969211357 wait complete. 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 Processor -- start 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- start start 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bd4102200 0x7f0bd4101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 0x7f0bd41001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bd4107460 0x7f0bd41006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd410d320 con 0x7f0bd4104b30 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd410d1a0 con 0x7f0bd4102200 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bdaffb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd410d4a0 con 0x7f0bd4107460 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bd3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 0x7f0bd41001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bd3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 0x7f0bd41001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37688/0 (socket says 172.21.15.16:37688) 2024-04-25T01:25:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bd3fff640 1 -- 172.21.15.16:0/3730301503 learned_addr learned my addr 172.21.15.16:0/3730301503 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.188+0000 7f0bd8d97640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bd4102200 0x7f0bd4101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd9598640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bd4107460 0x7f0bd41006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd3fff640 1 -- 172.21.15.16:0/3730301503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bd4107460 msgr2=0x7f0bd41006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd3fff640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bd4107460 0x7f0bd41006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd3fff640 1 -- 172.21.15.16:0/3730301503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bd4102200 msgr2=0x7f0bd4101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd3fff640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bd4102200 0x7f0bd4101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd3fff640 1 -- 172.21.15.16:0/3730301503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd4100f80 con 0x7f0bd4104b30 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd3fff640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 0x7f0bd41001b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800d740 tx=0x7f0bc800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8017070 con 0x7f0bd4104b30 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0bc800de50 con 0x7f0bd4104b30 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd41a1af0 con 0x7f0bd4104b30 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8004d60 con 0x7f0bd4104b30 2024-04-25T01:25:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.189+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd41a2040 con 0x7f0bd4104b30 2024-04-25T01:25:20.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.190+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd40fd6e0 con 0x7f0bd4104b30 2024-04-25T01:25:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.191+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0bc8004f00 con 0x7f0bd4104b30 2024-04-25T01:25:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.192+0000 7f0bd1ffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0bbc0a2c10 0x7f0bbc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.192+0000 7f0bd8d97640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0bbc0a2c10 0x7f0bbc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.192+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0bc80ce8e0 con 0x7f0bd4104b30 2024-04-25T01:25:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.192+0000 7f0bd8d97640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0bbc0a2c10 0x7f0bbc0a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009bd0 tx=0x7f0bc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:20.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.196+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0bc809b8f0 con 0x7f0bd4104b30 2024-04-25T01:25:20.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:20.422+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f0bd4062340 con 0x7f0bd4104b30 2024-04-25T01:25:21.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.064+0000 7f0bd1ffb640 1 -- 172.21.15.16:0/3730301503 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7f0bc809b1f0 con 0x7f0bd4104b30 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.066+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0bbc0a2c10 msgr2=0x7f0bbc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.066+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0bbc0a2c10 0x7f0bbc0a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009bd0 tx=0x7f0bc4005f90 comp rx=0 tx=0).stop 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.066+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 msgr2=0x7f0bd41001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.066+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 0x7f0bd41001b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800d740 tx=0x7f0bc800dc00 comp rx=0 tx=0).stop 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 shutdown_connections 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0bbc0a2c10 0x7f0bbc0a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bd4107460 0x7f0bd41006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bd4104b30 0x7f0bd41001b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 --2- 172.21.15.16:0/3730301503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0bd4102200 0x7f0bd4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 >> 172.21.15.16:0/3730301503 conn(0x7f0bd40fbbf0 msgr2=0x7f0bd4102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 shutdown_connections 2024-04-25T01:25:21.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.067+0000 7f0bdaffb640 1 -- 172.21.15.16:0/3730301503 wait complete. 2024-04-25T01:25:21.061 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-25T01:25:21.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 -- 172.21.15.16:0/3998370498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 msgr2=0x7fdc881098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 --2- 172.21.15.16:0/3998370498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 0x7fdc881098d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009ca0 tx=0x7fdc8002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 -- 172.21.15.16:0/3998370498 shutdown_connections 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 --2- 172.21.15.16:0/3998370498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 0x7fdc881098d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 --2- 172.21.15.16:0/3998370498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 0x7fdc88106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 --2- 172.21.15.16:0/3998370498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc88104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 -- 172.21.15.16:0/3998370498 >> 172.21.15.16:0/3998370498 conn(0x7fdc880fbc30 msgr2=0x7fdc880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 -- 172.21.15.16:0/3998370498 shutdown_connections 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.192+0000 7fdc8e209640 1 -- 172.21.15.16:0/3998370498 wait complete. 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 Processor -- start 2024-04-25T01:25:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 -- start start 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc881979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 0x7fdc88197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 0x7fdc8819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8810d2f0 con 0x7fdc88102220 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8810d170 con 0x7fdc88107480 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc8e209640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8810d470 con 0x7fdc88104b50 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.193+0000 7fdc877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc881979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc881979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37714/0 (socket says 172.21.15.16:37714) 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 -- 172.21.15.16:0/2902459198 learned_addr learned my addr 172.21.15.16:0/2902459198 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc87fff640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 0x7fdc8819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 0x7fdc88197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 -- 172.21.15.16:0/2902459198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 msgr2=0x7fdc88197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 0x7fdc88197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 -- 172.21.15.16:0/2902459198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 msgr2=0x7fdc8819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 0x7fdc8819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 -- 172.21.15.16:0/2902459198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8819f5e0 con 0x7fdc88102220 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 0x7fdc88197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc877fe640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc881979c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800e9f0 tx=0x7fdc7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7800cd80 con 0x7fdc88102220 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc780108e0 con 0x7fdc88102220 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8819f8d0 con 0x7fdc88102220 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78018920 con 0x7fdc88102220 2024-04-25T01:25:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.194+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc8819fe20 con 0x7fdc88102220 2024-04-25T01:25:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.195+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc880fd720 con 0x7fdc88102220 2024-04-25T01:25:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.196+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdc78010a90 con 0x7fdc88102220 2024-04-25T01:25:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.197+0000 7fdc84ff9640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdc700a2b90 0x7fdc700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.197+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdc700a2b90 0x7fdc700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.198+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fdc780cf270 con 0x7fdc88102220 2024-04-25T01:25:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.198+0000 7fdc86ffd640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdc700a2b90 0x7fdc700a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0058c0 tx=0x7fdc7c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.202+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc7809c320 con 0x7fdc88102220 2024-04-25T01:25:21.216 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:20 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: 2024-04-25T01:25:20.919+0000 7fed4d080700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:25:21.216 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:25:20 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: 2024-04-25T01:25:20.919+0000 7fed4d080700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T01:25:21.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: pgmap v295: 97 pgs: 67 active+clean, 21 creating+peering, 9 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-04-25T01:25:21.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/74823016' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:25:21.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] up:active 2024-04-25T01:25:21.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:25:21.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:25:21.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: Cluster is now healthy 2024-04-25T01:25:21.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:25:21.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:25:21.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:25:21.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:21 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3730301503' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: pgmap v295: 97 pgs: 67 active+clean, 21 creating+peering, 9 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/74823016' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] up:active 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: Cluster is now healthy 2024-04-25T01:25:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:25:21.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:25:21.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:25:21.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3730301503' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:25:21.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.413+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdc88061b80 con 0x7fdc700a2b90 2024-04-25T01:25:21.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.435+0000 7fdc84ff9640 1 -- 172.21.15.16:0/2902459198 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdc88061b80 con 0x7fdc700a2b90 2024-04-25T01:25:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdc700a2b90 msgr2=0x7fdc700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdc700a2b90 0x7fdc700a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0058c0 tx=0x7fdc7c005850 comp rx=0 tx=0).stop 2024-04-25T01:25:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 msgr2=0x7fdc881979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc881979c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800e9f0 tx=0x7fdc7800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 shutdown_connections 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdc700a2b90 0x7fdc700a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc88107480 0x7fdc8819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc88104b50 0x7fdc88197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 --2- 172.21.15.16:0/2902459198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc88102220 0x7fdc881979c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 >> 172.21.15.16:0/2902459198 conn(0x7fdc880fbc30 msgr2=0x7fdc88102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 shutdown_connections 2024-04-25T01:25:21.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.437+0000 7fdc8e209640 1 -- 172.21.15.16:0/2902459198 wait complete. 2024-04-25T01:25:21.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T01:25:21.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: pgmap v295: 97 pgs: 67 active+clean, 21 creating+peering, 9 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-04-25T01:25:21.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/74823016' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] up:active 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: Cluster is now healthy 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:25:21.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:21 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3730301503' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:25:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 -- 172.21.15.16:0/2299339723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 msgr2=0x7fbfa8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2299339723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8100ac0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009900 tx=0x7fbfa402a410 comp rx=0 tx=0).stop 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 -- 172.21.15.16:0/2299339723 shutdown_connections 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2299339723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfa8102250 0x7fbfa8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2299339723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa8101890 0x7fbfa8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2299339723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8100ac0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 -- 172.21.15.16:0/2299339723 >> 172.21.15.16:0/2299339723 conn(0x7fbfa80fbc30 msgr2=0x7fbfa80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 -- 172.21.15.16:0/2299339723 shutdown_connections 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 -- 172.21.15.16:0/2299339723 wait complete. 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 Processor -- start 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.555+0000 7fbfb0412640 1 -- start start 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa8101890 0x7fbfa819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfa8102250 0x7fbfa81a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa810c220 con 0x7fbfa81006a0 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa810c0a0 con 0x7fbfa8102250 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa810c3a0 con 0x7fbfa8101890 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37752/0 (socket says 172.21.15.16:37752) 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 -- 172.21.15.16:0/2075408708 learned_addr learned my addr 172.21.15.16:0/2075408708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae9af640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfa8102250 0x7fbfa81a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 -- 172.21.15.16:0/2075408708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa8101890 msgr2=0x7fbfa819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfad9ad640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa8101890 0x7fbfa819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa8101890 0x7fbfa819a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 -- 172.21.15.16:0/2075408708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfa8102250 msgr2=0x7fbfa81a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfa8102250 0x7fbfa81a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 -- 172.21.15.16:0/2075408708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa81a1900 con 0x7fbfa81006a0 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfae1ae640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8199cf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40314e0 tx=0x7fbfa4031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa403f030 con 0x7fbfa81006a0 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa40044b0 con 0x7fbfa81006a0 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa81a1b90 con 0x7fbfa81006a0 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4035070 con 0x7fbfa81006a0 2024-04-25T01:25:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.556+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa81a20d0 con 0x7fbfa81006a0 2024-04-25T01:25:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.557+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa81a22a0 con 0x7fbfa81006a0 2024-04-25T01:25:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.558+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbfa4042030 con 0x7fbfa81006a0 2024-04-25T01:25:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.559+0000 7fbf9f7fe640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf940a2c10 0x7fbf940a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.559+0000 7fbfad9ad640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf940a2c10 0x7fbf940a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.559+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbfa40ef8b0 con 0x7fbfa81006a0 2024-04-25T01:25:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.560+0000 7fbfad9ad640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf940a2c10 0x7fbf940a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800a810 tx=0x7fbf98008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.563+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbfa40bc940 con 0x7fbfa81006a0 2024-04-25T01:25:21.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.778+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbfa806a4a0 con 0x7fbf940a2c10 2024-04-25T01:25:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.803+0000 7fbf9f7fe640 1 -- 172.21.15.16:0/2075408708 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbfa806a4a0 con 0x7fbf940a2c10 2024-04-25T01:25:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf940a2c10 msgr2=0x7fbf940a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf940a2c10 0x7fbf940a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800a810 tx=0x7fbf98008040 comp rx=0 tx=0).stop 2024-04-25T01:25:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 msgr2=0x7fbfa8199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8199cf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40314e0 tx=0x7fbfa4031a30 comp rx=0 tx=0).stop 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 shutdown_connections 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf940a2c10 0x7fbf940a50d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfa8102250 0x7fbfa81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa8101890 0x7fbfa819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 --2- 172.21.15.16:0/2075408708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbfa81006a0 0x7fbfa8199cf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 >> 172.21.15.16:0/2075408708 conn(0x7fbfa80fbc30 msgr2=0x7fbfa8069d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 shutdown_connections 2024-04-25T01:25:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.805+0000 7fbfb0412640 1 -- 172.21.15.16:0/2075408708 wait complete. 2024-04-25T01:25:21.795 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:25:21.417+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:21.795 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:25:21.417+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:21.795 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:25:21.417+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:21.795 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:25:21.417+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:21.795 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:25:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:25:21.417+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:25:21.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T01:25:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 -- 172.21.15.16:0/2460506045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 msgr2=0x7f74501017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 --2- 172.21.15.16:0/2460506045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501017f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7448009ca0 tx=0x7f744802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 -- 172.21.15.16:0/2460506045 shutdown_connections 2024-04-25T01:25:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 --2- 172.21.15.16:0/2460506045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501017f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 --2- 172.21.15.16:0/2460506045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 0x7f7450100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 --2- 172.21.15.16:0/2460506045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7450100070 0x7f7450100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 -- 172.21.15.16:0/2460506045 >> 172.21.15.16:0/2460506045 conn(0x7f74500fbc30 msgr2=0x7f74500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 -- 172.21.15.16:0/2460506045 shutdown_connections 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 -- 172.21.15.16:0/2460506045 wait complete. 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.923+0000 7f74568ca640 1 Processor -- start 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 -- start start 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7450100070 0x7f7450199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 0x7f745019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f745010f4a0 con 0x7f7450101370 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f745010f320 con 0x7f74501009d0 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f74568ca640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f745010f620 con 0x7f7450100070 2024-04-25T01:25:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501a1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32824/0 (socket says 172.21.15.16:32824) 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 -- 172.21.15.16:0/2827328468 learned_addr learned my addr 172.21.15.16:0/2827328468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f744f7fe640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 0x7f745019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f744ffff640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7450100070 0x7f7450199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 -- 172.21.15.16:0/2827328468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7450100070 msgr2=0x7f7450199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7450100070 0x7f7450199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 -- 172.21.15.16:0/2827328468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 msgr2=0x7f745019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 0x7f745019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 -- 172.21.15.16:0/2827328468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74501a1790 con 0x7f7450101370 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f744f7fe640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 0x7f745019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.924+0000 7f7454e67640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501a1120 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7448031970 tx=0x7f7448003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.925+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448004040 con 0x7f7450101370 2024-04-25T01:25:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.925+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7448004690 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.925+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448030c60 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.925+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74501a1a20 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.925+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74501a1f20 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.926+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745010adc0 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.927+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f74480041e0 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.927+0000 7f744d7fa640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74380a2c10 0x7f74380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.928+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f74480f0500 con 0x7f7450101370 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.928+0000 7f744ffff640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74380a2c10 0x7f74380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.928+0000 7f744ffff640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74380a2c10 0x7f74380a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f744000bbe0 tx=0x7f744000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:21.931+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f74480bd5b0 con 0x7f7450101370 2024-04-25T01:25:22.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.088+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f74480b9520 con 0x7f7450101370 2024-04-25T01:25:22.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:22 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3730301503' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:25:22.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:22 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] up:active 2024-04-25T01:25:22.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:22 smithi016 ceph-mon[29153]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:25:22.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:22 smithi016 ceph-mon[29153]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:25:22.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:22 smithi016 ceph-mon[29153]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:22.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.152+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f7450061b80 con 0x7f74380a2c10 2024-04-25T01:25:22.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.157+0000 7f744d7fa640 1 -- 172.21.15.16:0/2827328468 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7450061b80 con 0x7f74380a2c10 2024-04-25T01:25:22.137 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/32ff4df1-a9e4-48d1-b4f8-1df2db566f01 2024-04-25T01:25:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74380a2c10 msgr2=0x7f74380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74380a2c10 0x7f74380a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f744000bbe0 tx=0x7f744000b390 comp rx=0 tx=0).stop 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 msgr2=0x7f74501a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501a1120 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7448031970 tx=0x7f7448003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 shutdown_connections 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74380a2c10 0x7f74380a50d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7450101370 0x7f74501a1120 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74501009d0 0x7f745019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 --2- 172.21.15.16:0/2827328468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7450100070 0x7f7450199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 >> 172.21.15.16:0/2827328468 conn(0x7f74500fbc30 msgr2=0x7f74500fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 shutdown_connections 2024-04-25T01:25:22.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.159+0000 7f74568ca640 1 -- 172.21.15.16:0/2827328468 wait complete. 2024-04-25T01:25:22.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T01:25:22.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:22 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3730301503' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:25:22.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:22 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] up:active 2024-04-25T01:25:22.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:22 smithi156 ceph-mon[32631]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:25:22.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:22 smithi156 ceph-mon[32631]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:25:22.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:22 smithi156 ceph-mon[32631]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 -- 172.21.15.16:0/3713764623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0100680 msgr2=0x7f80f0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 --2- 172.21.15.16:0/3713764623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0100680 0x7f80f0100aa0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009900 tx=0x7f80e002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 -- 172.21.15.16:0/3713764623 shutdown_connections 2024-04-25T01:25:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 --2- 172.21.15.16:0/3713764623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f0102230 0x7f80f0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 --2- 172.21.15.16:0/3713764623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0101870 0x7f80f0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 --2- 172.21.15.16:0/3713764623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0100680 0x7f80f0100aa0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 -- 172.21.15.16:0/3713764623 >> 172.21.15.16:0/3713764623 conn(0x7f80f00fbbf0 msgr2=0x7f80f00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 -- 172.21.15.16:0/3713764623 shutdown_connections 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.278+0000 7f80f487d640 1 -- 172.21.15.16:0/3713764623 wait complete. 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 Processor -- start 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 -- start start 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0100680 0x7f80f00694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f0101870 0x7f80f019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 0x7f80f019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010c180 con 0x7f80f0102230 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010c000 con 0x7f80f0101870 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80f487d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010c300 con 0x7f80f0100680 2024-04-25T01:25:22.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 0x7f80f019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 0x7f80f019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32840/0 (socket says 172.21.15.16:32840) 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 -- 172.21.15.16:0/444040699 learned_addr learned my addr 172.21.15.16:0/444040699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80edd9b640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f0101870 0x7f80f019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80ee59c640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0100680 0x7f80f00694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 -- 172.21.15.16:0/444040699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0100680 msgr2=0x7f80f00694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0100680 0x7f80f00694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 -- 172.21.15.16:0/444040699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f0101870 msgr2=0x7f80f019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f0101870 0x7f80f019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.279+0000 7f80eed9d640 1 -- 172.21.15.16:0/444040699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f019f650 con 0x7f80f0102230 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.280+0000 7f80eed9d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 0x7f80f019ed90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f80e800cad0 tx=0x7f80e800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.280+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e800ae20 con 0x7f80f0102230 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.280+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f019f940 con 0x7f80f0102230 2024-04-25T01:25:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.280+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e8005020 con 0x7f80f0102230 2024-04-25T01:25:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.280+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e800f6b0 con 0x7f80f0102230 2024-04-25T01:25:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.280+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f019fe90 con 0x7f80f0102230 2024-04-25T01:25:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.281+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f0107b60 con 0x7f80f0102230 2024-04-25T01:25:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.282+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f80e80051d0 con 0x7f80f0102230 2024-04-25T01:25:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.282+0000 7f80df7fe640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2c20 0x7f80d80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.283+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f80e80cf010 con 0x7f80f0102230 2024-04-25T01:25:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.283+0000 7f80ee59c640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2c20 0x7f80d80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.283+0000 7f80ee59c640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2c20 0x7f80d80a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f80e0000c00 tx=0x7f80e00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.287+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f80e8014030 con 0x7f80f0102230 2024-04-25T01:25:22.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:22 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3730301503' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:25:22.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:22 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] up:active 2024-04-25T01:25:22.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:22 smithi038 ceph-mon[32841]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:25:22.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:22 smithi038 ceph-mon[32841]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:25:22.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:22 smithi038 ceph-mon[32841]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:22.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.500+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f80f0061b80 con 0x7f80d80a2c20 2024-04-25T01:25:22.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.527+0000 7f80df7fe640 1 -- 172.21.15.16:0/444040699 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f80f0061b80 con 0x7f80d80a2c20 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2c20 msgr2=0x7f80d80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2c20 0x7f80d80a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f80e0000c00 tx=0x7f80e00314a0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 msgr2=0x7f80f019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 0x7f80f019ed90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f80e800cad0 tx=0x7f80e800cf90 comp rx=0 tx=0).stop 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 shutdown_connections 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2c20 0x7f80d80a50e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102230 0x7f80f019ed90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f0101870 0x7f80f019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 --2- 172.21.15.16:0/444040699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0100680 0x7f80f00694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 >> 172.21.15.16:0/444040699 conn(0x7f80f00fbbf0 msgr2=0x7f80f0104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 shutdown_connections 2024-04-25T01:25:22.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.529+0000 7f80f487d640 1 -- 172.21.15.16:0/444040699 wait complete. 2024-04-25T01:25:22.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 -- 172.21.15.16:0/644771769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 msgr2=0x7f2788106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 --2- 172.21.15.16:0/644771769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788106f20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f277c009900 tx=0x7f277c02a410 comp rx=0 tx=0).stop 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 -- 172.21.15.16:0/644771769 shutdown_connections 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 --2- 172.21.15.16:0/644771769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 0x7f27881098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 --2- 172.21.15.16:0/644771769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788106f20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 --2- 172.21.15.16:0/644771769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2788102200 0x7f27881045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.646+0000 7f278ebee640 1 -- 172.21.15.16:0/644771769 >> 172.21.15.16:0/644771769 conn(0x7f27880fbbf0 msgr2=0x7f27880fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 -- 172.21.15.16:0/644771769 shutdown_connections 2024-04-25T01:25:22.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 -- 172.21.15.16:0/644771769 wait complete. 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 Processor -- start 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 -- start start 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2788102200 0x7f27881979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 0x7f278819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f278810d2a0 con 0x7f2788104b30 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f278810d120 con 0x7f2788107460 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f278ebee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f278810d420 con 0x7f2788102200 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f2787fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f2787fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32866/0 (socket says 172.21.15.16:32866) 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.647+0000 7f2787fff640 1 -- 172.21.15.16:0/3225308300 learned_addr learned my addr 172.21.15.16:0/3225308300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:22.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f278c98a640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2788102200 0x7f27881979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f278d18b640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 0x7f278819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2787fff640 1 -- 172.21.15.16:0/3225308300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2788102200 msgr2=0x7f27881979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2787fff640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2788102200 0x7f27881979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2787fff640 1 -- 172.21.15.16:0/3225308300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 msgr2=0x7f278819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2787fff640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 0x7f278819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2787fff640 1 -- 172.21.15.16:0/3225308300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278819f5f0 con 0x7f2788104b30 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f278d18b640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 0x7f278819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2787fff640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788197f10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f277c0314e0 tx=0x7f277c0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c03f030 con 0x7f2788104b30 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278819f880 con 0x7f2788104b30 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f277c003cb0 con 0x7f2788104b30 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c035070 con 0x7f2788104b30 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.648+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278819fdc0 con 0x7f2788104b30 2024-04-25T01:25:22.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.649+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27880fd6e0 con 0x7f2788104b30 2024-04-25T01:25:22.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.650+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f277c042030 con 0x7f2788104b30 2024-04-25T01:25:22.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.651+0000 7f2785ffb640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f27700a2c20 0x7f27700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.651+0000 7f278c98a640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f27700a2c20 0x7f27700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.651+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f277c0ef710 con 0x7f2788104b30 2024-04-25T01:25:22.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.651+0000 7f278c98a640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f27700a2c20 0x7f27700a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2780007350 tx=0x7f27800072e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.655+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f277c0bc6c0 con 0x7f2788104b30 2024-04-25T01:25:22.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.869+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f2788061c10 con 0x7f27700a2c20 2024-04-25T01:25:22.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.874+0000 7f2785ffb640 1 -- 172.21.15.16:0/3225308300 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2788061c10 con 0x7f27700a2c20 2024-04-25T01:25:22.854 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/3a5f0d36-0795-4006-848f-fe34f8e87ca0 2024-04-25T01:25:22.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f27700a2c20 msgr2=0x7f27700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f27700a2c20 0x7f27700a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2780007350 tx=0x7f27800072e0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 msgr2=0x7f2788197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788197f10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f277c0314e0 tx=0x7f277c0042e0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 shutdown_connections 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f27700a2c20 0x7f27700a50e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2788107460 0x7f278819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104b30 0x7f2788197f10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 --2- 172.21.15.16:0/3225308300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2788102200 0x7f27881979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.876+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 >> 172.21.15.16:0/3225308300 conn(0x7f27880fbbf0 msgr2=0x7f2788102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.877+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 shutdown_connections 2024-04-25T01:25:22.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.877+0000 7f278ebee640 1 -- 172.21.15.16:0/3225308300 wait complete. 2024-04-25T01:25:22.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.989+0000 7f999de38640 1 -- 172.21.15.16:0/4003769694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981029a0 msgr2=0x7f9998102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.989+0000 7f999de38640 1 --2- 172.21.15.16:0/4003769694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981029a0 0x7f9998102e20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9980009900 tx=0x7f998002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 -- 172.21.15.16:0/4003769694 shutdown_connections 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 --2- 172.21.15.16:0/4003769694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9998103360 0x7f9998109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 --2- 172.21.15.16:0/4003769694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981029a0 0x7f9998102e20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 --2- 172.21.15.16:0/4003769694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99981017b0 0x7f9998101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 -- 172.21.15.16:0/4003769694 >> 172.21.15.16:0/4003769694 conn(0x7f99980fcd40 msgr2=0x7f99980ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 -- 172.21.15.16:0/4003769694 shutdown_connections 2024-04-25T01:25:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 -- 172.21.15.16:0/4003769694 wait complete. 2024-04-25T01:25:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 Processor -- start 2024-04-25T01:25:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.990+0000 7f999de38640 1 -- start start 2024-04-25T01:25:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 0x7f99980754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99981029a0 0x7f9998073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9998103360 0x7f9998074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f999810d2e0 con 0x7f99981017b0 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f999810d160 con 0x7f9998103360 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f999810d460 con 0x7f99981029a0 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 0x7f99980754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 0x7f99980754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32884/0 (socket says 172.21.15.16:32884) 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 -- 172.21.15.16:0/2782064559 learned_addr learned my addr 172.21.15.16:0/2782064559 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f9996ffd640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99981029a0 0x7f9998073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 -- 172.21.15.16:0/2782064559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99981029a0 msgr2=0x7f9998073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99981029a0 0x7f9998073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 -- 172.21.15.16:0/2782064559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9998103360 msgr2=0x7f9998074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9998103360 0x7f9998074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 -- 172.21.15.16:0/2782064559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9998074900 con 0x7f99981017b0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f99977fe640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 0x7f99980754f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9988009fd0 tx=0x7f998800c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988011470 con 0x7f99981017b0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9988011ac0 con 0x7f99981017b0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99981a1ad0 con 0x7f99981017b0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998801aab0 con 0x7f99981017b0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.991+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99981a2020 con 0x7f99981017b0 2024-04-25T01:25:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.992+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9998108b70 con 0x7f99981017b0 2024-04-25T01:25:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.993+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9988011c70 con 0x7f99981017b0 2024-04-25T01:25:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.994+0000 7f9994ff9640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f99840a2c20 0x7f99840a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.994+0000 7f9996ffd640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f99840a2c20 0x7f99840a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.994+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f99880cf6b0 con 0x7f99981017b0 2024-04-25T01:25:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.994+0000 7f9996ffd640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f99840a2c20 0x7f99840a50e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9998075260 tx=0x7f99800314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:22.998+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9988016030 con 0x7f99981017b0 2024-04-25T01:25:23.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.221+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f999802d170 con 0x7f99981017b0 2024-04-25T01:25:23.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.222+0000 7f9994ff9640 1 -- 172.21.15.16:0/2782064559 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11866 (secure 0 0 0) 0x7f999802d170 con 0x7f99981017b0 2024-04-25T01:25:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:23.202 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:25:21.068325+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24476,24499,24524]},"id":1}]} 2024-04-25T01:25:23.202 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:25:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f99840a2c20 msgr2=0x7f99840a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f99840a2c20 0x7f99840a50e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9998075260 tx=0x7f99800314a0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 msgr2=0x7f99980754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 0x7f99980754f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9988009fd0 tx=0x7f998800c270 comp rx=0 tx=0).stop 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 shutdown_connections 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f99840a2c20 0x7f99840a50e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9998103360 0x7f9998074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99981029a0 0x7f9998073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 --2- 172.21.15.16:0/2782064559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99981017b0 0x7f99980754f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 >> 172.21.15.16:0/2782064559 conn(0x7f99980fcd40 msgr2=0x7f9998105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 shutdown_connections 2024-04-25T01:25:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.224+0000 7f999de38640 1 -- 172.21.15.16:0/2782064559 wait complete. 2024-04-25T01:25:23.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:23 smithi016 ceph-mon[29153]: pgmap v296: 97 pgs: 93 active+clean, 4 creating+peering; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 3.5 KiB/s wr, 9 op/s 2024-04-25T01:25:23.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:23 smithi016 ceph-mon[29153]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:23 smithi016 ceph-mon[29153]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:25:23.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:23 smithi016 ceph-mon[29153]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:23 smithi016 ceph-mon[29153]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:25:23.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 -- 172.21.15.16:0/3200141202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f301006a0 msgr2=0x7f2f30100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3200141202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f301006a0 0x7f2f30100ac0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009900 tx=0x7f2f2002a410 comp rx=0 tx=0).stop 2024-04-25T01:25:23.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 -- 172.21.15.16:0/3200141202 shutdown_connections 2024-04-25T01:25:23.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3200141202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f30102250 0x7f2f30108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3200141202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f30101890 0x7f2f30101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3200141202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f301006a0 0x7f2f30100ac0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 -- 172.21.15.16:0/3200141202 >> 172.21.15.16:0/3200141202 conn(0x7f2f300fbc30 msgr2=0x7f2f300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 -- 172.21.15.16:0/3200141202 shutdown_connections 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 -- 172.21.15.16:0/3200141202 wait complete. 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.337+0000 7f2f366ba640 1 Processor -- start 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 -- start start 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f301006a0 0x7f2f3006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f30101890 0x7f2f3019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 0x7f2f3019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3010c1a0 con 0x7f2f30102250 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3010c020 con 0x7f2f301006a0 2024-04-25T01:25:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f366ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3010c320 con 0x7f2f30101890 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 0x7f2f3019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 0x7f2f3019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32900/0 (socket says 172.21.15.16:32900) 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 -- 172.21.15.16:0/3163627776 learned_addr learned my addr 172.21.15.16:0/3163627776 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f301006a0 0x7f2f3006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f2f7fe640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f30101890 0x7f2f3019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 -- 172.21.15.16:0/3163627776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f30101890 msgr2=0x7f2f3019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f30101890 0x7f2f3019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 -- 172.21.15.16:0/3163627776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f301006a0 msgr2=0x7f2f3006a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f301006a0 0x7f2f3006a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 -- 172.21.15.16:0/3163627776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3019f650 con 0x7f2f30102250 2024-04-25T01:25:23.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.338+0000 7f2f34c57640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 0x7f2f3019ed90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800cad0 tx=0x7f2f2800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.339+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800ae20 con 0x7f2f30102250 2024-04-25T01:25:23.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.339+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f28005020 con 0x7f2f30102250 2024-04-25T01:25:23.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.339+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3019f940 con 0x7f2f30102250 2024-04-25T01:25:23.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.339+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800f6b0 con 0x7f2f30102250 2024-04-25T01:25:23.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.339+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3019fe40 con 0x7f2f30102250 2024-04-25T01:25:23.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.340+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f30107ae0 con 0x7f2f30102250 2024-04-25T01:25:23.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.341+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2f280051d0 con 0x7f2f30102250 2024-04-25T01:25:23.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.341+0000 7f2f2d7fa640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2f180a2c20 0x7f2f180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.342+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2f280cf0c0 con 0x7f2f30102250 2024-04-25T01:25:23.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.342+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2f180a2c20 0x7f2f180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.342+0000 7f2f2ffff640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2f180a2c20 0x7f2f180a50e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2f20004770 tx=0x7f2f20000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.346+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2f28014030 con 0x7f2f30102250 2024-04-25T01:25:23.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:23 smithi038 ceph-mon[32841]: pgmap v296: 97 pgs: 93 active+clean, 4 creating+peering; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 3.5 KiB/s wr, 9 op/s 2024-04-25T01:25:23.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:23 smithi038 ceph-mon[32841]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:23 smithi038 ceph-mon[32841]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:25:23.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:23 smithi038 ceph-mon[32841]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:23 smithi038 ceph-mon[32841]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:23 smithi156 ceph-mon[32631]: pgmap v296: 97 pgs: 93 active+clean, 4 creating+peering; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 3.5 KiB/s wr, 9 op/s 2024-04-25T01:25:23.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:23 smithi156 ceph-mon[32631]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:23 smithi156 ceph-mon[32631]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:25:23.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:23 smithi156 ceph-mon[32631]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:23 smithi156 ceph-mon[32631]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.530+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f3002d390 con 0x7f2f30102250 2024-04-25T01:25:23.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.532+0000 7f2f2d7fa640 1 -- 172.21.15.16:0/3163627776 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22196 (secure 0 0 0) 0x7f2f2809c070 con 0x7f2f30102250 2024-04-25T01:25:23.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:23.512 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:25:17.723944+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2f180a2c20 msgr2=0x7f2f180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2f180a2c20 0x7f2f180a50e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2f20004770 tx=0x7f2f20000bc0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 msgr2=0x7f2f3019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 0x7f2f3019ed90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800cad0 tx=0x7f2f2800cf90 comp rx=0 tx=0).stop 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 shutdown_connections 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2f180a2c20 0x7f2f180a50e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f30102250 0x7f2f3019ed90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f30101890 0x7f2f3019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 --2- 172.21.15.16:0/3163627776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f301006a0 0x7f2f3006a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 >> 172.21.15.16:0/3163627776 conn(0x7f2f300fbc30 msgr2=0x7f2f30104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 shutdown_connections 2024-04-25T01:25:23.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.534+0000 7f2f366ba640 1 -- 172.21.15.16:0/3163627776 wait complete. 2024-04-25T01:25:23.525 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:25:23.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:25:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 -- 172.21.15.16:0/2329394636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 msgr2=0x7f4544108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 --2- 172.21.15.16:0/2329394636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f4544108ae0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f453c009ca0 tx=0x7f453c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 -- 172.21.15.16:0/2329394636 shutdown_connections 2024-04-25T01:25:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 --2- 172.21.15.16:0/2329394636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f4544108ae0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 --2- 172.21.15.16:0/2329394636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4544101890 0x7f4544101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 --2- 172.21.15.16:0/2329394636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45441006a0 0x7f4544100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.647+0000 7f454c56a640 1 -- 172.21.15.16:0/2329394636 >> 172.21.15.16:0/2329394636 conn(0x7f45440fbc30 msgr2=0x7f45440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 -- 172.21.15.16:0/2329394636 shutdown_connections 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 -- 172.21.15.16:0/2329394636 wait complete. 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 Processor -- start 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 -- start start 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45441006a0 0x7f4544197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544101890 0x7f4544197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f454419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f454410c2a0 con 0x7f4544102250 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f454410c120 con 0x7f45441006a0 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454c56a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f454410c420 con 0x7f4544101890 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454ab07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f454419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454ab07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f454419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32920/0 (socket says 172.21.15.16:32920) 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454ab07640 1 -- 172.21.15.16:0/306024163 learned_addr learned my addr 172.21.15.16:0/306024163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f454a306640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45441006a0 0x7f4544197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.648+0000 7f4549b05640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544101890 0x7f4544197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454ab07640 1 -- 172.21.15.16:0/306024163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544101890 msgr2=0x7f4544197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454ab07640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544101890 0x7f4544197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454ab07640 1 -- 172.21.15.16:0/306024163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45441006a0 msgr2=0x7f4544197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454ab07640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45441006a0 0x7f4544197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454ab07640 1 -- 172.21.15.16:0/306024163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454419f660 con 0x7f4544102250 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454ab07640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f454419eff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f453c031040 tx=0x7f453c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c032ec0 con 0x7f4544102250 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454419f8f0 con 0x7f4544102250 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f453c0044f0 con 0x7f4544102250 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c030c30 con 0x7f4544102250 2024-04-25T01:25:23.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.649+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f454419fe40 con 0x7f4544102250 2024-04-25T01:25:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.650+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45441a0010 con 0x7f4544102250 2024-04-25T01:25:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.651+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f453c004040 con 0x7f4544102250 2024-04-25T01:25:23.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.652+0000 7f453b7fe640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f45300a2dc0 0x7f45300a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.652+0000 7f454a306640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f45300a2dc0 0x7f45300a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.652+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f453c0f0520 con 0x7f4544102250 2024-04-25T01:25:23.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.652+0000 7f454a306640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f45300a2dc0 0x7f45300a5280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4540005fd0 tx=0x7f454000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.656+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f453c0bd5e0 con 0x7f4544102250 2024-04-25T01:25:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.838+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f45441a01e0 con 0x7f45300a2dc0 2024-04-25T01:25:23.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.845+0000 7f453b7fe640 1 -- 172.21.15.16:0/306024163 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258894 (secure 0 0 0) 0x7f45441a01e0 con 0x7f45300a2dc0 2024-04-25T01:25:23.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-25T01:25:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f45300a2dc0 msgr2=0x7f45300a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f45300a2dc0 0x7f45300a5280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4540005fd0 tx=0x7f454000b390 comp rx=0 tx=0).stop 2024-04-25T01:25:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 msgr2=0x7f454419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f454419eff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f453c031040 tx=0x7f453c003ba0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 shutdown_connections 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f45300a2dc0 0x7f45300a5280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4544102250 0x7f454419eff0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544101890 0x7f4544197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 --2- 172.21.15.16:0/306024163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45441006a0 0x7f4544197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 >> 172.21.15.16:0/306024163 conn(0x7f45440fbc30 msgr2=0x7f45440fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 shutdown_connections 2024-04-25T01:25:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:23.849+0000 7f454c56a640 1 -- 172.21.15.16:0/306024163 wait complete. 2024-04-25T01:25:23.840 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-04-25T01:25:23.610350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594968,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":141,"num_read_kb":121,"num_write":294,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368620,"kb_used_data":11112,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573204,"statfs":{"total":1151940427776,"available":1151562960896,"internally_reserved":0,"allocated":11378688,"data_stored":7104031,"data_compressed":111527,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":15,"num_read_kb":12,"num_write":113,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.647625"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957164+0000","last_change":"2024-04-25T01:25:16.740766+0000","last_active":"2024-04-25T01:25:17.957164+0000","last_peered":"2024-04-25T01:25:17.957164+0000","last_clean":"2024-04-25T01:25:17.957164+0000","last_became_active":"2024-04-25T01:25:16.740181+0000","last_became_peered":"2024-04-25T01:25:16.740181+0000","last_unstale":"2024-04-25T01:25:17.957164+0000","last_undegraded":"2024-04-25T01:25:17.957164+0000","last_fullsized":"2024-04-25T01:25:17.957164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:59.154389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955972+0000","last_change":"2024-04-25T01:25:16.750724+0000","last_active":"2024-04-25T01:25:17.955972+0000","last_peered":"2024-04-25T01:25:17.955972+0000","last_clean":"2024-04-25T01:25:17.955972+0000","last_became_active":"2024-04-25T01:25:16.750391+0000","last_became_peered":"2024-04-25T01:25:16.750391+0000","last_unstale":"2024-04-25T01:25:17.955972+0000","last_undegraded":"2024-04-25T01:25:17.955972+0000","last_fullsized":"2024-04-25T01:25:17.955972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:15.576020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956219+0000","last_change":"2024-04-25T01:25:16.753914+0000","last_active":"2024-04-25T01:25:17.956219+0000","last_peered":"2024-04-25T01:25:17.956219+0000","last_clean":"2024-04-25T01:25:17.956219+0000","last_became_active":"2024-04-25T01:25:16.753464+0000","last_became_peered":"2024-04-25T01:25:16.753464+0000","last_unstale":"2024-04-25T01:25:17.956219+0000","last_undegraded":"2024-04-25T01:25:17.956219+0000","last_fullsized":"2024-04-25T01:25:17.956219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:55.716948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957081+0000","last_change":"2024-04-25T01:25:16.754586+0000","last_active":"2024-04-25T01:25:17.957081+0000","last_peered":"2024-04-25T01:25:17.957081+0000","last_clean":"2024-04-25T01:25:17.957081+0000","last_became_active":"2024-04-25T01:25:16.752260+0000","last_became_peered":"2024-04-25T01:25:16.752260+0000","last_unstale":"2024-04-25T01:25:17.957081+0000","last_undegraded":"2024-04-25T01:25:17.957081+0000","last_fullsized":"2024-04-25T01:25:17.957081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:30.935184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956682+0000","last_change":"2024-04-25T01:25:16.741397+0000","last_active":"2024-04-25T01:25:17.956682+0000","last_peered":"2024-04-25T01:25:17.956682+0000","last_clean":"2024-04-25T01:25:17.956682+0000","last_became_active":"2024-04-25T01:25:16.741056+0000","last_became_peered":"2024-04-25T01:25:16.741056+0000","last_unstale":"2024-04-25T01:25:17.956682+0000","last_undegraded":"2024-04-25T01:25:17.956682+0000","last_fullsized":"2024-04-25T01:25:17.956682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:38.886023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737037+0000","last_change":"2024-04-25T01:25:16.757351+0000","last_active":"2024-04-25T01:25:17.737037+0000","last_peered":"2024-04-25T01:25:17.737037+0000","last_clean":"2024-04-25T01:25:17.737037+0000","last_became_active":"2024-04-25T01:25:16.756980+0000","last_became_peered":"2024-04-25T01:25:16.756980+0000","last_unstale":"2024-04-25T01:25:17.737037+0000","last_undegraded":"2024-04-25T01:25:17.737037+0000","last_fullsized":"2024-04-25T01:25:17.737037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:18.755302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541243+0000","last_change":"2024-04-25T01:25:16.740265+0000","last_active":"2024-04-25T01:25:18.541243+0000","last_peered":"2024-04-25T01:25:18.541243+0000","last_clean":"2024-04-25T01:25:18.541243+0000","last_became_active":"2024-04-25T01:25:16.739345+0000","last_became_peered":"2024-04-25T01:25:16.739345+0000","last_unstale":"2024-04-25T01:25:18.541243+0000","last_undegraded":"2024-04-25T01:25:18.541243+0000","last_fullsized":"2024-04-25T01:25:18.541243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:55.399650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955587+0000","last_change":"2024-04-25T01:25:16.753159+0000","last_active":"2024-04-25T01:25:17.955587+0000","last_peered":"2024-04-25T01:25:17.955587+0000","last_clean":"2024-04-25T01:25:17.955587+0000","last_became_active":"2024-04-25T01:25:16.751863+0000","last_became_peered":"2024-04-25T01:25:16.751863+0000","last_unstale":"2024-04-25T01:25:17.955587+0000","last_undegraded":"2024-04-25T01:25:17.955587+0000","last_fullsized":"2024-04-25T01:25:17.955587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:54.218096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957134+0000","last_change":"2024-04-25T01:25:16.738727+0000","last_active":"2024-04-25T01:25:17.957134+0000","last_peered":"2024-04-25T01:25:17.957134+0000","last_clean":"2024-04-25T01:25:17.957134+0000","last_became_active":"2024-04-25T01:25:16.738390+0000","last_became_peered":"2024-04-25T01:25:16.738390+0000","last_unstale":"2024-04-25T01:25:17.957134+0000","last_undegraded":"2024-04-25T01:25:17.957134+0000","last_fullsized":"2024-04-25T01:25:17.957134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:03.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737383+0000","last_change":"2024-04-25T01:25:16.756974+0000","last_active":"2024-04-25T01:25:17.737383+0000","last_peered":"2024-04-25T01:25:17.737383+0000","last_clean":"2024-04-25T01:25:17.737383+0000","last_became_active":"2024-04-25T01:25:16.756549+0000","last_became_peered":"2024-04-25T01:25:16.756549+0000","last_unstale":"2024-04-25T01:25:17.737383+0000","last_undegraded":"2024-04-25T01:25:17.737383+0000","last_fullsized":"2024-04-25T01:25:17.737383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:49.042132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541532+0000","last_change":"2024-04-25T01:25:16.745728+0000","last_active":"2024-04-25T01:25:18.541532+0000","last_peered":"2024-04-25T01:25:18.541532+0000","last_clean":"2024-04-25T01:25:18.541532+0000","last_became_active":"2024-04-25T01:25:16.745091+0000","last_became_peered":"2024-04-25T01:25:16.745091+0000","last_unstale":"2024-04-25T01:25:18.541532+0000","last_undegraded":"2024-04-25T01:25:18.541532+0000","last_fullsized":"2024-04-25T01:25:18.541532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:01.622335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957017+0000","last_change":"2024-04-25T01:25:16.755146+0000","last_active":"2024-04-25T01:25:17.957017+0000","last_peered":"2024-04-25T01:25:17.957017+0000","last_clean":"2024-04-25T01:25:17.957017+0000","last_became_active":"2024-04-25T01:25:16.752713+0000","last_became_peered":"2024-04-25T01:25:16.752713+0000","last_unstale":"2024-04-25T01:25:17.957017+0000","last_undegraded":"2024-04-25T01:25:17.957017+0000","last_fullsized":"2024-04-25T01:25:17.957017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:31.967903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731442+0000","last_change":"2024-04-25T01:25:16.751162+0000","last_active":"2024-04-25T01:25:17.731442+0000","last_peered":"2024-04-25T01:25:17.731442+0000","last_clean":"2024-04-25T01:25:17.731442+0000","last_became_active":"2024-04-25T01:25:16.750772+0000","last_became_peered":"2024-04-25T01:25:16.750772+0000","last_unstale":"2024-04-25T01:25:17.731442+0000","last_undegraded":"2024-04-25T01:25:17.731442+0000","last_fullsized":"2024-04-25T01:25:17.731442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:39.533120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955886+0000","last_change":"2024-04-25T01:25:16.754367+0000","last_active":"2024-04-25T01:25:17.955886+0000","last_peered":"2024-04-25T01:25:17.955886+0000","last_clean":"2024-04-25T01:25:17.955886+0000","last_became_active":"2024-04-25T01:25:16.750324+0000","last_became_peered":"2024-04-25T01:25:16.750324+0000","last_unstale":"2024-04-25T01:25:17.955886+0000","last_undegraded":"2024-04-25T01:25:17.955886+0000","last_fullsized":"2024-04-25T01:25:17.955886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:17.606187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955617+0000","last_change":"2024-04-25T01:25:16.753111+0000","last_active":"2024-04-25T01:25:17.955617+0000","last_peered":"2024-04-25T01:25:17.955617+0000","last_clean":"2024-04-25T01:25:17.955617+0000","last_became_active":"2024-04-25T01:25:16.752749+0000","last_became_peered":"2024-04-25T01:25:16.752749+0000","last_unstale":"2024-04-25T01:25:17.955617+0000","last_undegraded":"2024-04-25T01:25:17.955617+0000","last_fullsized":"2024-04-25T01:25:17.955617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:07:28.653568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956504+0000","last_change":"2024-04-25T01:25:16.744958+0000","last_active":"2024-04-25T01:25:17.956504+0000","last_peered":"2024-04-25T01:25:17.956504+0000","last_clean":"2024-04-25T01:25:17.956504+0000","last_became_active":"2024-04-25T01:25:16.744606+0000","last_became_peered":"2024-04-25T01:25:16.744606+0000","last_unstale":"2024-04-25T01:25:17.956504+0000","last_undegraded":"2024-04-25T01:25:17.956504+0000","last_fullsized":"2024-04-25T01:25:17.956504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:07:58.920942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957454+0000","last_change":"2024-04-25T01:25:16.743462+0000","last_active":"2024-04-25T01:25:17.957454+0000","last_peered":"2024-04-25T01:25:17.957454+0000","last_clean":"2024-04-25T01:25:17.957454+0000","last_became_active":"2024-04-25T01:25:16.743055+0000","last_became_peered":"2024-04-25T01:25:16.743055+0000","last_unstale":"2024-04-25T01:25:17.957454+0000","last_undegraded":"2024-04-25T01:25:17.957454+0000","last_fullsized":"2024-04-25T01:25:17.957454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:23:03.660065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956737+0000","last_change":"2024-04-25T01:25:16.751446+0000","last_active":"2024-04-25T01:25:17.956737+0000","last_peered":"2024-04-25T01:25:17.956737+0000","last_clean":"2024-04-25T01:25:17.956737+0000","last_became_active":"2024-04-25T01:25:16.751107+0000","last_became_peered":"2024-04-25T01:25:16.751107+0000","last_unstale":"2024-04-25T01:25:17.956737+0000","last_undegraded":"2024-04-25T01:25:17.956737+0000","last_fullsized":"2024-04-25T01:25:17.956737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:52.275662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.732141+0000","last_change":"2024-04-25T01:25:16.748478+0000","last_active":"2024-04-25T01:25:17.732141+0000","last_peered":"2024-04-25T01:25:17.732141+0000","last_clean":"2024-04-25T01:25:17.732141+0000","last_became_active":"2024-04-25T01:25:16.748064+0000","last_became_peered":"2024-04-25T01:25:16.748064+0000","last_unstale":"2024-04-25T01:25:17.732141+0000","last_undegraded":"2024-04-25T01:25:17.732141+0000","last_fullsized":"2024-04-25T01:25:17.732141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:45.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956794+0000","last_change":"2024-04-25T01:25:15.987611+0000","last_active":"2024-04-25T01:25:17.956794+0000","last_peered":"2024-04-25T01:25:17.956794+0000","last_clean":"2024-04-25T01:25:17.956794+0000","last_became_active":"2024-04-25T01:25:15.987300+0000","last_became_peered":"2024-04-25T01:25:15.987300+0000","last_unstale":"2024-04-25T01:25:17.956794+0000","last_undegraded":"2024-04-25T01:25:17.956794+0000","last_fullsized":"2024-04-25T01:25:17.956794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:24.636607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955696+0000","last_change":"2024-04-25T01:25:16.751076+0000","last_active":"2024-04-25T01:25:17.955696+0000","last_peered":"2024-04-25T01:25:17.955696+0000","last_clean":"2024-04-25T01:25:17.955696+0000","last_became_active":"2024-04-25T01:25:16.750219+0000","last_became_peered":"2024-04-25T01:25:16.750219+0000","last_unstale":"2024-04-25T01:25:17.955696+0000","last_undegraded":"2024-04-25T01:25:17.955696+0000","last_fullsized":"2024-04-25T01:25:17.955696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:11.956004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956263+0000","last_change":"2024-04-25T01:25:15.977434+0000","last_active":"2024-04-25T01:25:17.956263+0000","last_peered":"2024-04-25T01:25:17.956263+0000","last_clean":"2024-04-25T01:25:17.956263+0000","last_became_active":"2024-04-25T01:25:15.977095+0000","last_became_peered":"2024-04-25T01:25:15.977095+0000","last_unstale":"2024-04-25T01:25:17.956263+0000","last_undegraded":"2024-04-25T01:25:17.956263+0000","last_fullsized":"2024-04-25T01:25:17.956263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:01.102962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956536+0000","last_change":"2024-04-25T01:25:16.740061+0000","last_active":"2024-04-25T01:25:17.956536+0000","last_peered":"2024-04-25T01:25:17.956536+0000","last_clean":"2024-04-25T01:25:17.956536+0000","last_became_active":"2024-04-25T01:25:16.739593+0000","last_became_peered":"2024-04-25T01:25:16.739593+0000","last_unstale":"2024-04-25T01:25:17.956536+0000","last_undegraded":"2024-04-25T01:25:17.956536+0000","last_fullsized":"2024-04-25T01:25:17.956536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:32.711066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.071581+0000","last_change":"2024-04-25T01:25:15.984150+0000","last_active":"2024-04-25T01:25:20.071581+0000","last_peered":"2024-04-25T01:25:20.071581+0000","last_clean":"2024-04-25T01:25:20.071581+0000","last_became_active":"2024-04-25T01:25:15.983405+0000","last_became_peered":"2024-04-25T01:25:15.983405+0000","last_unstale":"2024-04-25T01:25:20.071581+0000","last_undegraded":"2024-04-25T01:25:20.071581+0000","last_fullsized":"2024-04-25T01:25:20.071581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:38.511851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956437+0000","last_change":"2024-04-25T01:25:16.751549+0000","last_active":"2024-04-25T01:25:17.956437+0000","last_peered":"2024-04-25T01:25:17.956437+0000","last_clean":"2024-04-25T01:25:17.956437+0000","last_became_active":"2024-04-25T01:25:16.751167+0000","last_became_peered":"2024-04-25T01:25:16.751167+0000","last_unstale":"2024-04-25T01:25:17.956437+0000","last_undegraded":"2024-04-25T01:25:17.956437+0000","last_fullsized":"2024-04-25T01:25:17.956437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:21.260019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957152+0000","last_change":"2024-04-25T01:25:15.977916+0000","last_active":"2024-04-25T01:25:17.957152+0000","last_peered":"2024-04-25T01:25:17.957152+0000","last_clean":"2024-04-25T01:25:17.957152+0000","last_became_active":"2024-04-25T01:25:15.977602+0000","last_became_peered":"2024-04-25T01:25:15.977602+0000","last_unstale":"2024-04-25T01:25:17.957152+0000","last_undegraded":"2024-04-25T01:25:17.957152+0000","last_fullsized":"2024-04-25T01:25:17.957152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:27.926489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957490+0000","last_change":"2024-04-25T01:25:16.756728+0000","last_active":"2024-04-25T01:25:17.957490+0000","last_peered":"2024-04-25T01:25:17.957490+0000","last_clean":"2024-04-25T01:25:17.957490+0000","last_became_active":"2024-04-25T01:25:16.756363+0000","last_became_peered":"2024-04-25T01:25:16.756363+0000","last_unstale":"2024-04-25T01:25:17.957490+0000","last_undegraded":"2024-04-25T01:25:17.957490+0000","last_fullsized":"2024-04-25T01:25:17.957490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:06.582831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:22.079357+0000","last_change":"2024-04-25T01:25:15.987099+0000","last_active":"2024-04-25T01:25:22.079357+0000","last_peered":"2024-04-25T01:25:22.079357+0000","last_clean":"2024-04-25T01:25:22.079357+0000","last_became_active":"2024-04-25T01:25:15.986745+0000","last_became_peered":"2024-04-25T01:25:15.986745+0000","last_unstale":"2024-04-25T01:25:22.079357+0000","last_undegraded":"2024-04-25T01:25:22.079357+0000","last_fullsized":"2024-04-25T01:25:22.079357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:14.944342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956643+0000","last_change":"2024-04-25T01:25:16.742735+0000","last_active":"2024-04-25T01:25:17.956643+0000","last_peered":"2024-04-25T01:25:17.956643+0000","last_clean":"2024-04-25T01:25:17.956643+0000","last_became_active":"2024-04-25T01:25:16.742391+0000","last_became_peered":"2024-04-25T01:25:16.742391+0000","last_unstale":"2024-04-25T01:25:17.956643+0000","last_undegraded":"2024-04-25T01:25:17.956643+0000","last_fullsized":"2024-04-25T01:25:17.956643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:05.783111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073249+0000","last_change":"2024-04-25T01:25:15.982605+0000","last_active":"2024-04-25T01:25:20.073249+0000","last_peered":"2024-04-25T01:25:20.073249+0000","last_clean":"2024-04-25T01:25:20.073249+0000","last_became_active":"2024-04-25T01:25:15.982264+0000","last_became_peered":"2024-04-25T01:25:15.982264+0000","last_unstale":"2024-04-25T01:25:20.073249+0000","last_undegraded":"2024-04-25T01:25:20.073249+0000","last_fullsized":"2024-04-25T01:25:20.073249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:09.960827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957619+0000","last_change":"2024-04-25T01:25:16.747784+0000","last_active":"2024-04-25T01:25:17.957619+0000","last_peered":"2024-04-25T01:25:17.957619+0000","last_clean":"2024-04-25T01:25:17.957619+0000","last_became_active":"2024-04-25T01:25:16.747432+0000","last_became_peered":"2024-04-25T01:25:16.747432+0000","last_unstale":"2024-04-25T01:25:17.957619+0000","last_undegraded":"2024-04-25T01:25:17.957619+0000","last_fullsized":"2024-04-25T01:25:17.957619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:40.602582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.074441+0000","last_change":"2024-04-25T01:25:15.977370+0000","last_active":"2024-04-25T01:25:20.074441+0000","last_peered":"2024-04-25T01:25:20.074441+0000","last_clean":"2024-04-25T01:25:20.074441+0000","last_became_active":"2024-04-25T01:25:15.977010+0000","last_became_peered":"2024-04-25T01:25:15.977010+0000","last_unstale":"2024-04-25T01:25:20.074441+0000","last_undegraded":"2024-04-25T01:25:20.074441+0000","last_fullsized":"2024-04-25T01:25:20.074441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:41.239154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955636+0000","last_change":"2024-04-25T01:25:16.741490+0000","last_active":"2024-04-25T01:25:17.955636+0000","last_peered":"2024-04-25T01:25:17.955636+0000","last_clean":"2024-04-25T01:25:17.955636+0000","last_became_active":"2024-04-25T01:25:16.741148+0000","last_became_peered":"2024-04-25T01:25:16.741148+0000","last_unstale":"2024-04-25T01:25:17.955636+0000","last_undegraded":"2024-04-25T01:25:17.955636+0000","last_fullsized":"2024-04-25T01:25:17.955636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:50.918575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.440350+0000","last_change":"2024-04-25T01:25:15.988283+0000","last_active":"2024-04-25T01:25:18.440350+0000","last_peered":"2024-04-25T01:25:18.440350+0000","last_clean":"2024-04-25T01:25:18.440350+0000","last_became_active":"2024-04-25T01:25:15.987894+0000","last_became_peered":"2024-04-25T01:25:15.987894+0000","last_unstale":"2024-04-25T01:25:18.440350+0000","last_undegraded":"2024-04-25T01:25:18.440350+0000","last_fullsized":"2024-04-25T01:25:18.440350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:01.220170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956377+0000","last_change":"2024-04-25T01:25:16.749468+0000","last_active":"2024-04-25T01:25:17.956377+0000","last_peered":"2024-04-25T01:25:17.956377+0000","last_clean":"2024-04-25T01:25:17.956377+0000","last_became_active":"2024-04-25T01:25:16.749094+0000","last_became_peered":"2024-04-25T01:25:16.749094+0000","last_unstale":"2024-04-25T01:25:17.956377+0000","last_undegraded":"2024-04-25T01:25:17.956377+0000","last_fullsized":"2024-04-25T01:25:17.956377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:17.701149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.067512+0000","last_change":"2024-04-25T01:25:15.981590+0000","last_active":"2024-04-25T01:25:19.067512+0000","last_peered":"2024-04-25T01:25:19.067512+0000","last_clean":"2024-04-25T01:25:19.067512+0000","last_became_active":"2024-04-25T01:25:15.981192+0000","last_became_peered":"2024-04-25T01:25:15.981192+0000","last_unstale":"2024-04-25T01:25:19.067512+0000","last_undegraded":"2024-04-25T01:25:19.067512+0000","last_fullsized":"2024-04-25T01:25:19.067512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:06.220365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956930+0000","last_change":"2024-04-25T01:25:16.751414+0000","last_active":"2024-04-25T01:25:17.956930+0000","last_peered":"2024-04-25T01:25:17.956930+0000","last_clean":"2024-04-25T01:25:17.956930+0000","last_became_active":"2024-04-25T01:25:16.751003+0000","last_became_peered":"2024-04-25T01:25:16.751003+0000","last_unstale":"2024-04-25T01:25:17.956930+0000","last_undegraded":"2024-04-25T01:25:17.956930+0000","last_fullsized":"2024-04-25T01:25:17.956930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:49:03.146685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068928+0000","last_change":"2024-04-25T01:25:15.974041+0000","last_active":"2024-04-25T01:25:19.068928+0000","last_peered":"2024-04-25T01:25:19.068928+0000","last_clean":"2024-04-25T01:25:19.068928+0000","last_became_active":"2024-04-25T01:25:15.973621+0000","last_became_peered":"2024-04-25T01:25:15.973621+0000","last_unstale":"2024-04-25T01:25:19.068928+0000","last_undegraded":"2024-04-25T01:25:19.068928+0000","last_fullsized":"2024-04-25T01:25:19.068928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:12.039161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956496+0000","last_change":"2024-04-25T01:25:16.745295+0000","last_active":"2024-04-25T01:25:17.956496+0000","last_peered":"2024-04-25T01:25:17.956496+0000","last_clean":"2024-04-25T01:25:17.956496+0000","last_became_active":"2024-04-25T01:25:16.744931+0000","last_became_peered":"2024-04-25T01:25:16.744931+0000","last_unstale":"2024-04-25T01:25:17.956496+0000","last_undegraded":"2024-04-25T01:25:17.956496+0000","last_fullsized":"2024-04-25T01:25:17.956496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:48.993537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.067512+0000","last_change":"2024-04-25T01:25:15.980900+0000","last_active":"2024-04-25T01:25:19.067512+0000","last_peered":"2024-04-25T01:25:19.067512+0000","last_clean":"2024-04-25T01:25:19.067512+0000","last_became_active":"2024-04-25T01:25:15.980528+0000","last_became_peered":"2024-04-25T01:25:15.980528+0000","last_unstale":"2024-04-25T01:25:19.067512+0000","last_undegraded":"2024-04-25T01:25:19.067512+0000","last_fullsized":"2024-04-25T01:25:19.067512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:47.743151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956549+0000","last_change":"2024-04-25T01:25:16.753886+0000","last_active":"2024-04-25T01:25:17.956549+0000","last_peered":"2024-04-25T01:25:17.956549+0000","last_clean":"2024-04-25T01:25:17.956549+0000","last_became_active":"2024-04-25T01:25:16.753522+0000","last_became_peered":"2024-04-25T01:25:16.753522+0000","last_unstale":"2024-04-25T01:25:17.956549+0000","last_undegraded":"2024-04-25T01:25:17.956549+0000","last_fullsized":"2024-04-25T01:25:17.956549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:12:46.103249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.065934+0000","last_change":"2024-04-25T01:25:15.980114+0000","last_active":"2024-04-25T01:25:19.065934+0000","last_peered":"2024-04-25T01:25:19.065934+0000","last_clean":"2024-04-25T01:25:19.065934+0000","last_became_active":"2024-04-25T01:25:15.979676+0000","last_became_peered":"2024-04-25T01:25:15.979676+0000","last_unstale":"2024-04-25T01:25:19.065934+0000","last_undegraded":"2024-04-25T01:25:19.065934+0000","last_fullsized":"2024-04-25T01:25:19.065934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:07.401375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956419+0000","last_change":"2024-04-25T01:25:16.739726+0000","last_active":"2024-04-25T01:25:17.956419+0000","last_peered":"2024-04-25T01:25:17.956419+0000","last_clean":"2024-04-25T01:25:17.956419+0000","last_became_active":"2024-04-25T01:25:16.739399+0000","last_became_peered":"2024-04-25T01:25:16.739399+0000","last_unstale":"2024-04-25T01:25:17.956419+0000","last_undegraded":"2024-04-25T01:25:17.956419+0000","last_fullsized":"2024-04-25T01:25:17.956419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:21:42.747061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.072422+0000","last_change":"2024-04-25T01:25:15.976731+0000","last_active":"2024-04-25T01:25:20.072422+0000","last_peered":"2024-04-25T01:25:20.072422+0000","last_clean":"2024-04-25T01:25:20.072422+0000","last_became_active":"2024-04-25T01:25:15.976405+0000","last_became_peered":"2024-04-25T01:25:15.976405+0000","last_unstale":"2024-04-25T01:25:20.072422+0000","last_undegraded":"2024-04-25T01:25:20.072422+0000","last_fullsized":"2024-04-25T01:25:20.072422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:21.612389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731512+0000","last_change":"2024-04-25T01:25:16.741378+0000","last_active":"2024-04-25T01:25:17.731512+0000","last_peered":"2024-04-25T01:25:17.731512+0000","last_clean":"2024-04-25T01:25:17.731512+0000","last_became_active":"2024-04-25T01:25:16.741066+0000","last_became_peered":"2024-04-25T01:25:16.741066+0000","last_unstale":"2024-04-25T01:25:17.731512+0000","last_undegraded":"2024-04-25T01:25:17.731512+0000","last_fullsized":"2024-04-25T01:25:17.731512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:37.288181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068861+0000","last_change":"2024-04-25T01:25:15.988288+0000","last_active":"2024-04-25T01:25:19.068861+0000","last_peered":"2024-04-25T01:25:19.068861+0000","last_clean":"2024-04-25T01:25:19.068861+0000","last_became_active":"2024-04-25T01:25:15.987860+0000","last_became_peered":"2024-04-25T01:25:15.987860+0000","last_unstale":"2024-04-25T01:25:19.068861+0000","last_undegraded":"2024-04-25T01:25:19.068861+0000","last_fullsized":"2024-04-25T01:25:19.068861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:23.634199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541710+0000","last_change":"2024-04-25T01:25:16.743784+0000","last_active":"2024-04-25T01:25:18.541710+0000","last_peered":"2024-04-25T01:25:18.541710+0000","last_clean":"2024-04-25T01:25:18.541710+0000","last_became_active":"2024-04-25T01:25:16.743435+0000","last_became_peered":"2024-04-25T01:25:16.743435+0000","last_unstale":"2024-04-25T01:25:18.541710+0000","last_undegraded":"2024-04-25T01:25:18.541710+0000","last_fullsized":"2024-04-25T01:25:18.541710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:52:08.297617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956027+0000","last_change":"2024-04-25T01:25:15.978201+0000","last_active":"2024-04-25T01:25:17.956027+0000","last_peered":"2024-04-25T01:25:17.956027+0000","last_clean":"2024-04-25T01:25:17.956027+0000","last_became_active":"2024-04-25T01:25:15.977832+0000","last_became_peered":"2024-04-25T01:25:15.977832+0000","last_unstale":"2024-04-25T01:25:17.956027+0000","last_undegraded":"2024-04-25T01:25:17.956027+0000","last_fullsized":"2024-04-25T01:25:17.956027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:42.483969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541659+0000","last_change":"2024-04-25T01:25:16.750944+0000","last_active":"2024-04-25T01:25:18.541659+0000","last_peered":"2024-04-25T01:25:18.541659+0000","last_clean":"2024-04-25T01:25:18.541659+0000","last_became_active":"2024-04-25T01:25:16.750588+0000","last_became_peered":"2024-04-25T01:25:16.750588+0000","last_unstale":"2024-04-25T01:25:18.541659+0000","last_undegraded":"2024-04-25T01:25:18.541659+0000","last_fullsized":"2024-04-25T01:25:18.541659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:31.473184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.075003+0000","last_change":"2024-04-25T01:25:15.976656+0000","last_active":"2024-04-25T01:25:20.075003+0000","last_peered":"2024-04-25T01:25:20.075003+0000","last_clean":"2024-04-25T01:25:20.075003+0000","last_became_active":"2024-04-25T01:25:15.976318+0000","last_became_peered":"2024-04-25T01:25:15.976318+0000","last_unstale":"2024-04-25T01:25:20.075003+0000","last_undegraded":"2024-04-25T01:25:20.075003+0000","last_fullsized":"2024-04-25T01:25:20.075003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:49.702258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956906+0000","last_change":"2024-04-25T01:25:16.748178+0000","last_active":"2024-04-25T01:25:17.956906+0000","last_peered":"2024-04-25T01:25:17.956906+0000","last_clean":"2024-04-25T01:25:17.956906+0000","last_became_active":"2024-04-25T01:25:16.747868+0000","last_became_peered":"2024-04-25T01:25:16.747868+0000","last_unstale":"2024-04-25T01:25:17.956906+0000","last_undegraded":"2024-04-25T01:25:17.956906+0000","last_fullsized":"2024-04-25T01:25:17.956906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:06.378387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:21.068756+0000","last_change":"2024-04-25T01:25:15.981651+0000","last_active":"2024-04-25T01:25:21.068756+0000","last_peered":"2024-04-25T01:25:21.068756+0000","last_clean":"2024-04-25T01:25:21.068756+0000","last_became_active":"2024-04-25T01:25:15.981245+0000","last_became_peered":"2024-04-25T01:25:15.981245+0000","last_unstale":"2024-04-25T01:25:21.068756+0000","last_undegraded":"2024-04-25T01:25:21.068756+0000","last_fullsized":"2024-04-25T01:25:21.068756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:20.009347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956478+0000","last_change":"2024-04-25T01:25:16.756873+0000","last_active":"2024-04-25T01:25:17.956478+0000","last_peered":"2024-04-25T01:25:17.956478+0000","last_clean":"2024-04-25T01:25:17.956478+0000","last_became_active":"2024-04-25T01:25:16.756505+0000","last_became_peered":"2024-04-25T01:25:16.756505+0000","last_unstale":"2024-04-25T01:25:17.956478+0000","last_undegraded":"2024-04-25T01:25:17.956478+0000","last_fullsized":"2024-04-25T01:25:17.956478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:23.412303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956708+0000","last_change":"2024-04-25T01:25:15.983640+0000","last_active":"2024-04-25T01:25:17.956708+0000","last_peered":"2024-04-25T01:25:17.956708+0000","last_clean":"2024-04-25T01:25:17.956708+0000","last_became_active":"2024-04-25T01:25:15.983280+0000","last_became_peered":"2024-04-25T01:25:15.983280+0000","last_unstale":"2024-04-25T01:25:17.956708+0000","last_undegraded":"2024-04-25T01:25:17.956708+0000","last_fullsized":"2024-04-25T01:25:17.956708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:18.493139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737350+0000","last_change":"2024-04-25T01:25:16.751616+0000","last_active":"2024-04-25T01:25:17.737350+0000","last_peered":"2024-04-25T01:25:17.737350+0000","last_clean":"2024-04-25T01:25:17.737350+0000","last_became_active":"2024-04-25T01:25:16.751254+0000","last_became_peered":"2024-04-25T01:25:16.751254+0000","last_unstale":"2024-04-25T01:25:17.737350+0000","last_undegraded":"2024-04-25T01:25:17.737350+0000","last_fullsized":"2024-04-25T01:25:17.737350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:55.556054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541568+0000","last_change":"2024-04-25T01:25:15.977774+0000","last_active":"2024-04-25T01:25:18.541568+0000","last_peered":"2024-04-25T01:25:18.541568+0000","last_clean":"2024-04-25T01:25:18.541568+0000","last_became_active":"2024-04-25T01:25:15.977398+0000","last_became_peered":"2024-04-25T01:25:15.977398+0000","last_unstale":"2024-04-25T01:25:18.541568+0000","last_undegraded":"2024-04-25T01:25:18.541568+0000","last_fullsized":"2024-04-25T01:25:18.541568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:33.729091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737011+0000","last_change":"2024-04-25T01:25:16.749604+0000","last_active":"2024-04-25T01:25:17.737011+0000","last_peered":"2024-04-25T01:25:17.737011+0000","last_clean":"2024-04-25T01:25:17.737011+0000","last_became_active":"2024-04-25T01:25:16.749236+0000","last_became_peered":"2024-04-25T01:25:16.749236+0000","last_unstale":"2024-04-25T01:25:17.737011+0000","last_undegraded":"2024-04-25T01:25:17.737011+0000","last_fullsized":"2024-04-25T01:25:17.737011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:20:54.362813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957283+0000","last_change":"2024-04-25T01:25:16.746447+0000","last_active":"2024-04-25T01:25:17.957283+0000","last_peered":"2024-04-25T01:25:17.957283+0000","last_clean":"2024-04-25T01:25:17.957283+0000","last_became_active":"2024-04-25T01:25:16.745978+0000","last_became_peered":"2024-04-25T01:25:16.745978+0000","last_unstale":"2024-04-25T01:25:17.957283+0000","last_undegraded":"2024-04-25T01:25:17.957283+0000","last_fullsized":"2024-04-25T01:25:17.957283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:41.249662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073887+0000","last_change":"2024-04-25T01:25:15.987083+0000","last_active":"2024-04-25T01:25:20.073887+0000","last_peered":"2024-04-25T01:25:20.073887+0000","last_clean":"2024-04-25T01:25:20.073887+0000","last_became_active":"2024-04-25T01:25:15.986632+0000","last_became_peered":"2024-04-25T01:25:15.986632+0000","last_unstale":"2024-04-25T01:25:20.073887+0000","last_undegraded":"2024-04-25T01:25:20.073887+0000","last_fullsized":"2024-04-25T01:25:20.073887+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:44.792239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956209+0000","last_change":"2024-04-25T01:25:16.742007+0000","last_active":"2024-04-25T01:25:17.956209+0000","last_peered":"2024-04-25T01:25:17.956209+0000","last_clean":"2024-04-25T01:25:17.956209+0000","last_became_active":"2024-04-25T01:25:16.741464+0000","last_became_peered":"2024-04-25T01:25:16.741464+0000","last_unstale":"2024-04-25T01:25:17.956209+0000","last_undegraded":"2024-04-25T01:25:17.956209+0000","last_fullsized":"2024-04-25T01:25:17.956209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:32.171648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068452+0000","last_change":"2024-04-25T01:25:15.978255+0000","last_active":"2024-04-25T01:25:19.068452+0000","last_peered":"2024-04-25T01:25:19.068452+0000","last_clean":"2024-04-25T01:25:19.068452+0000","last_became_active":"2024-04-25T01:25:15.977860+0000","last_became_peered":"2024-04-25T01:25:15.977860+0000","last_unstale":"2024-04-25T01:25:19.068452+0000","last_undegraded":"2024-04-25T01:25:19.068452+0000","last_fullsized":"2024-04-25T01:25:19.068452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.516137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":339,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541038+0000","last_change":"2024-04-25T01:23:58.204099+0000","last_active":"2024-04-25T01:25:18.541038+0000","last_peered":"2024-04-25T01:25:18.541038+0000","last_clean":"2024-04-25T01:25:18.541038+0000","last_became_active":"2024-04-25T01:23:58.203533+0000","last_became_peered":"2024-04-25T01:23:58.203533+0000","last_unstale":"2024-04-25T01:25:18.541038+0000","last_undegraded":"2024-04-25T01:25:18.541038+0000","last_fullsized":"2024-04-25T01:25:18.541038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_clean_scrub_stamp":"2024-04-25T01:20:28.584758+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:17.504360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731154+0000","last_change":"2024-04-25T01:25:16.746335+0000","last_active":"2024-04-25T01:25:17.731154+0000","last_peered":"2024-04-25T01:25:17.731154+0000","last_clean":"2024-04-25T01:25:17.731154+0000","last_became_active":"2024-04-25T01:25:16.745878+0000","last_became_peered":"2024-04-25T01:25:16.745878+0000","last_unstale":"2024-04-25T01:25:17.731154+0000","last_undegraded":"2024-04-25T01:25:17.731154+0000","last_fullsized":"2024-04-25T01:25:17.731154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:15:35.318542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073285+0000","last_change":"2024-04-25T01:25:15.986400+0000","last_active":"2024-04-25T01:25:20.073285+0000","last_peered":"2024-04-25T01:25:20.073285+0000","last_clean":"2024-04-25T01:25:20.073285+0000","last_became_active":"2024-04-25T01:25:15.985975+0000","last_became_peered":"2024-04-25T01:25:15.985975+0000","last_unstale":"2024-04-25T01:25:20.073285+0000","last_undegraded":"2024-04-25T01:25:20.073285+0000","last_fullsized":"2024-04-25T01:25:20.073285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:12.666394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.065873+0000","last_change":"2024-04-25T01:25:15.981370+0000","last_active":"2024-04-25T01:25:19.065873+0000","last_peered":"2024-04-25T01:25:19.065873+0000","last_clean":"2024-04-25T01:25:19.065873+0000","last_became_active":"2024-04-25T01:25:15.980995+0000","last_became_peered":"2024-04-25T01:25:15.980995+0000","last_unstale":"2024-04-25T01:25:19.065873+0000","last_undegraded":"2024-04-25T01:25:19.065873+0000","last_fullsized":"2024-04-25T01:25:19.065873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:29.754441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731403+0000","last_change":"2024-04-25T01:25:16.744887+0000","last_active":"2024-04-25T01:25:17.731403+0000","last_peered":"2024-04-25T01:25:17.731403+0000","last_clean":"2024-04-25T01:25:17.731403+0000","last_became_active":"2024-04-25T01:25:16.744538+0000","last_became_peered":"2024-04-25T01:25:16.744538+0000","last_unstale":"2024-04-25T01:25:17.731403+0000","last_undegraded":"2024-04-25T01:25:17.731403+0000","last_fullsized":"2024-04-25T01:25:17.731403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:32.132521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.048072+0000","last_change":"2024-04-25T01:25:15.990181+0000","last_active":"2024-04-25T01:25:19.048072+0000","last_peered":"2024-04-25T01:25:19.048072+0000","last_clean":"2024-04-25T01:25:19.048072+0000","last_became_active":"2024-04-25T01:25:15.989816+0000","last_became_peered":"2024-04-25T01:25:15.989816+0000","last_unstale":"2024-04-25T01:25:19.048072+0000","last_undegraded":"2024-04-25T01:25:19.048072+0000","last_fullsized":"2024-04-25T01:25:19.048072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:17:58.377057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956438+0000","last_change":"2024-04-25T01:25:16.746199+0000","last_active":"2024-04-25T01:25:17.956438+0000","last_peered":"2024-04-25T01:25:17.956438+0000","last_clean":"2024-04-25T01:25:17.956438+0000","last_became_active":"2024-04-25T01:25:16.745857+0000","last_became_peered":"2024-04-25T01:25:16.745857+0000","last_unstale":"2024-04-25T01:25:17.956438+0000","last_undegraded":"2024-04-25T01:25:17.956438+0000","last_fullsized":"2024-04-25T01:25:17.956438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:21.392309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068066+0000","last_change":"2024-04-25T01:25:15.981173+0000","last_active":"2024-04-25T01:25:19.068066+0000","last_peered":"2024-04-25T01:25:19.068066+0000","last_clean":"2024-04-25T01:25:19.068066+0000","last_became_active":"2024-04-25T01:25:15.980718+0000","last_became_peered":"2024-04-25T01:25:15.980718+0000","last_unstale":"2024-04-25T01:25:19.068066+0000","last_undegraded":"2024-04-25T01:25:19.068066+0000","last_fullsized":"2024-04-25T01:25:19.068066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:54.086543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956129+0000","last_change":"2024-04-25T01:25:16.753987+0000","last_active":"2024-04-25T01:25:17.956129+0000","last_peered":"2024-04-25T01:25:17.956129+0000","last_clean":"2024-04-25T01:25:17.956129+0000","last_became_active":"2024-04-25T01:25:16.753546+0000","last_became_peered":"2024-04-25T01:25:16.753546+0000","last_unstale":"2024-04-25T01:25:17.956129+0000","last_undegraded":"2024-04-25T01:25:17.956129+0000","last_fullsized":"2024-04-25T01:25:17.956129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:10.318501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068770+0000","last_change":"2024-04-25T01:25:15.980938+0000","last_active":"2024-04-25T01:25:19.068770+0000","last_peered":"2024-04-25T01:25:19.068770+0000","last_clean":"2024-04-25T01:25:19.068770+0000","last_became_active":"2024-04-25T01:25:15.980586+0000","last_became_peered":"2024-04-25T01:25:15.980586+0000","last_unstale":"2024-04-25T01:25:19.068770+0000","last_undegraded":"2024-04-25T01:25:19.068770+0000","last_fullsized":"2024-04-25T01:25:19.068770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:00.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956369+0000","last_change":"2024-04-25T01:25:16.749711+0000","last_active":"2024-04-25T01:25:17.956369+0000","last_peered":"2024-04-25T01:25:17.956369+0000","last_clean":"2024-04-25T01:25:17.956369+0000","last_became_active":"2024-04-25T01:25:16.749218+0000","last_became_peered":"2024-04-25T01:25:16.749218+0000","last_unstale":"2024-04-25T01:25:17.956369+0000","last_undegraded":"2024-04-25T01:25:17.956369+0000","last_fullsized":"2024-04-25T01:25:17.956369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:56.555204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.067510+0000","last_change":"2024-04-25T01:25:15.988257+0000","last_active":"2024-04-25T01:25:19.067510+0000","last_peered":"2024-04-25T01:25:19.067510+0000","last_clean":"2024-04-25T01:25:19.067510+0000","last_became_active":"2024-04-25T01:25:15.987469+0000","last_became_peered":"2024-04-25T01:25:15.987469+0000","last_unstale":"2024-04-25T01:25:19.067510+0000","last_undegraded":"2024-04-25T01:25:19.067510+0000","last_fullsized":"2024-04-25T01:25:19.067510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:29.849868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956742+0000","last_change":"2024-04-25T01:25:16.746686+0000","last_active":"2024-04-25T01:25:17.956742+0000","last_peered":"2024-04-25T01:25:17.956742+0000","last_clean":"2024-04-25T01:25:17.956742+0000","last_became_active":"2024-04-25T01:25:16.746028+0000","last_became_peered":"2024-04-25T01:25:16.746028+0000","last_unstale":"2024-04-25T01:25:17.956742+0000","last_undegraded":"2024-04-25T01:25:17.956742+0000","last_fullsized":"2024-04-25T01:25:17.956742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:34:35.172207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731990+0000","last_change":"2024-04-25T01:25:15.974569+0000","last_active":"2024-04-25T01:25:17.731990+0000","last_peered":"2024-04-25T01:25:17.731990+0000","last_clean":"2024-04-25T01:25:17.731990+0000","last_became_active":"2024-04-25T01:25:15.973668+0000","last_became_peered":"2024-04-25T01:25:15.973668+0000","last_unstale":"2024-04-25T01:25:17.731990+0000","last_undegraded":"2024-04-25T01:25:17.731990+0000","last_fullsized":"2024-04-25T01:25:17.731990+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:02.813912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956677+0000","last_change":"2024-04-25T01:25:16.752510+0000","last_active":"2024-04-25T01:25:17.956677+0000","last_peered":"2024-04-25T01:25:17.956677+0000","last_clean":"2024-04-25T01:25:17.956677+0000","last_became_active":"2024-04-25T01:25:16.751889+0000","last_became_peered":"2024-04-25T01:25:16.751889+0000","last_unstale":"2024-04-25T01:25:17.956677+0000","last_undegraded":"2024-04-25T01:25:17.956677+0000","last_fullsized":"2024-04-25T01:25:17.956677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:44.628980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.069039+0000","last_change":"2024-04-25T01:25:15.983638+0000","last_active":"2024-04-25T01:25:19.069039+0000","last_peered":"2024-04-25T01:25:19.069039+0000","last_clean":"2024-04-25T01:25:19.069039+0000","last_became_active":"2024-04-25T01:25:15.983293+0000","last_became_peered":"2024-04-25T01:25:15.983293+0000","last_unstale":"2024-04-25T01:25:19.069039+0000","last_undegraded":"2024-04-25T01:25:19.069039+0000","last_fullsized":"2024-04-25T01:25:19.069039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:04:27.495446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541613+0000","last_change":"2024-04-25T01:25:16.736420+0000","last_active":"2024-04-25T01:25:18.541613+0000","last_peered":"2024-04-25T01:25:18.541613+0000","last_clean":"2024-04-25T01:25:18.541613+0000","last_became_active":"2024-04-25T01:25:16.736031+0000","last_became_peered":"2024-04-25T01:25:16.736031+0000","last_unstale":"2024-04-25T01:25:18.541613+0000","last_undegraded":"2024-04-25T01:25:18.541613+0000","last_fullsized":"2024-04-25T01:25:18.541613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:33.380905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073477+0000","last_change":"2024-04-25T01:25:15.982201+0000","last_active":"2024-04-25T01:25:20.073477+0000","last_peered":"2024-04-25T01:25:20.073477+0000","last_clean":"2024-04-25T01:25:20.073477+0000","last_became_active":"2024-04-25T01:25:15.981820+0000","last_became_peered":"2024-04-25T01:25:15.981820+0000","last_unstale":"2024-04-25T01:25:20.073477+0000","last_undegraded":"2024-04-25T01:25:20.073477+0000","last_fullsized":"2024-04-25T01:25:20.073477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:12.149417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541612+0000","last_change":"2024-04-25T01:25:16.739126+0000","last_active":"2024-04-25T01:25:18.541612+0000","last_peered":"2024-04-25T01:25:18.541612+0000","last_clean":"2024-04-25T01:25:18.541612+0000","last_became_active":"2024-04-25T01:25:16.738762+0000","last_became_peered":"2024-04-25T01:25:16.738762+0000","last_unstale":"2024-04-25T01:25:18.541612+0000","last_undegraded":"2024-04-25T01:25:18.541612+0000","last_fullsized":"2024-04-25T01:25:18.541612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:00:47.661399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.440463+0000","last_change":"2024-04-25T01:25:15.985732+0000","last_active":"2024-04-25T01:25:18.440463+0000","last_peered":"2024-04-25T01:25:18.440463+0000","last_clean":"2024-04-25T01:25:18.440463+0000","last_became_active":"2024-04-25T01:25:15.985300+0000","last_became_peered":"2024-04-25T01:25:15.985300+0000","last_unstale":"2024-04-25T01:25:18.440463+0000","last_undegraded":"2024-04-25T01:25:18.440463+0000","last_fullsized":"2024-04-25T01:25:18.440463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:01.180165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541598+0000","last_change":"2024-04-25T01:25:16.746639+0000","last_active":"2024-04-25T01:25:18.541598+0000","last_peered":"2024-04-25T01:25:18.541598+0000","last_clean":"2024-04-25T01:25:18.541598+0000","last_became_active":"2024-04-25T01:25:16.746291+0000","last_became_peered":"2024-04-25T01:25:16.746291+0000","last_unstale":"2024-04-25T01:25:18.541598+0000","last_undegraded":"2024-04-25T01:25:18.541598+0000","last_fullsized":"2024-04-25T01:25:18.541598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:31.346243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731564+0000","last_change":"2024-04-25T01:25:15.989193+0000","last_active":"2024-04-25T01:25:17.731564+0000","last_peered":"2024-04-25T01:25:17.731564+0000","last_clean":"2024-04-25T01:25:17.731564+0000","last_became_active":"2024-04-25T01:25:15.988811+0000","last_became_peered":"2024-04-25T01:25:15.988811+0000","last_unstale":"2024-04-25T01:25:17.731564+0000","last_undegraded":"2024-04-25T01:25:17.731564+0000","last_fullsized":"2024-04-25T01:25:17.731564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:56.030398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541730+0000","last_change":"2024-04-25T01:25:16.742209+0000","last_active":"2024-04-25T01:25:18.541730+0000","last_peered":"2024-04-25T01:25:18.541730+0000","last_clean":"2024-04-25T01:25:18.541730+0000","last_became_active":"2024-04-25T01:25:16.741810+0000","last_became_peered":"2024-04-25T01:25:16.741810+0000","last_unstale":"2024-04-25T01:25:18.541730+0000","last_undegraded":"2024-04-25T01:25:18.541730+0000","last_fullsized":"2024-04-25T01:25:18.541730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:38.008108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956617+0000","last_change":"2024-04-25T01:25:16.747264+0000","last_active":"2024-04-25T01:25:17.956617+0000","last_peered":"2024-04-25T01:25:17.956617+0000","last_clean":"2024-04-25T01:25:17.956617+0000","last_became_active":"2024-04-25T01:25:16.746514+0000","last_became_peered":"2024-04-25T01:25:16.746514+0000","last_unstale":"2024-04-25T01:25:17.956617+0000","last_undegraded":"2024-04-25T01:25:17.956617+0000","last_fullsized":"2024-04-25T01:25:17.956617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:14.672062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.732307+0000","last_change":"2024-04-25T01:25:16.753382+0000","last_active":"2024-04-25T01:25:17.732307+0000","last_peered":"2024-04-25T01:25:17.732307+0000","last_clean":"2024-04-25T01:25:17.732307+0000","last_became_active":"2024-04-25T01:25:16.752907+0000","last_became_peered":"2024-04-25T01:25:16.752907+0000","last_unstale":"2024-04-25T01:25:17.732307+0000","last_undegraded":"2024-04-25T01:25:17.732307+0000","last_fullsized":"2024-04-25T01:25:17.732307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:28.184630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731975+0000","last_change":"2024-04-25T01:25:16.752000+0000","last_active":"2024-04-25T01:25:17.731975+0000","last_peered":"2024-04-25T01:25:17.731975+0000","last_clean":"2024-04-25T01:25:17.731975+0000","last_became_active":"2024-04-25T01:25:16.749443+0000","last_became_peered":"2024-04-25T01:25:16.749443+0000","last_unstale":"2024-04-25T01:25:17.731975+0000","last_undegraded":"2024-04-25T01:25:17.731975+0000","last_fullsized":"2024-04-25T01:25:17.731975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:02.287719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956047+0000","last_change":"2024-04-25T01:25:16.742510+0000","last_active":"2024-04-25T01:25:17.956047+0000","last_peered":"2024-04-25T01:25:17.956047+0000","last_clean":"2024-04-25T01:25:17.956047+0000","last_became_active":"2024-04-25T01:25:16.742162+0000","last_became_peered":"2024-04-25T01:25:16.742162+0000","last_unstale":"2024-04-25T01:25:17.956047+0000","last_undegraded":"2024-04-25T01:25:17.956047+0000","last_fullsized":"2024-04-25T01:25:17.956047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:02.924505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731635+0000","last_change":"2024-04-25T01:25:16.756224+0000","last_active":"2024-04-25T01:25:17.731635+0000","last_peered":"2024-04-25T01:25:17.731635+0000","last_clean":"2024-04-25T01:25:17.731635+0000","last_became_active":"2024-04-25T01:25:16.755868+0000","last_became_peered":"2024-04-25T01:25:16.755868+0000","last_unstale":"2024-04-25T01:25:17.731635+0000","last_undegraded":"2024-04-25T01:25:17.731635+0000","last_fullsized":"2024-04-25T01:25:17.731635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:53.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956887+0000","last_change":"2024-04-25T01:25:16.745406+0000","last_active":"2024-04-25T01:25:17.956887+0000","last_peered":"2024-04-25T01:25:17.956887+0000","last_clean":"2024-04-25T01:25:17.956887+0000","last_became_active":"2024-04-25T01:25:16.744972+0000","last_became_peered":"2024-04-25T01:25:16.744972+0000","last_unstale":"2024-04-25T01:25:17.956887+0000","last_undegraded":"2024-04-25T01:25:17.956887+0000","last_fullsized":"2024-04-25T01:25:17.956887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:13:31.598070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.732364+0000","last_change":"2024-04-25T01:25:16.745294+0000","last_active":"2024-04-25T01:25:17.732364+0000","last_peered":"2024-04-25T01:25:17.732364+0000","last_clean":"2024-04-25T01:25:17.732364+0000","last_became_active":"2024-04-25T01:25:16.743285+0000","last_became_peered":"2024-04-25T01:25:16.743285+0000","last_unstale":"2024-04-25T01:25:17.732364+0000","last_undegraded":"2024-04-25T01:25:17.732364+0000","last_fullsized":"2024-04-25T01:25:17.732364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:36.189339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955773+0000","last_change":"2024-04-25T01:25:16.755167+0000","last_active":"2024-04-25T01:25:17.955773+0000","last_peered":"2024-04-25T01:25:17.955773+0000","last_clean":"2024-04-25T01:25:17.955773+0000","last_became_active":"2024-04-25T01:25:16.754787+0000","last_became_peered":"2024-04-25T01:25:16.754787+0000","last_unstale":"2024-04-25T01:25:17.955773+0000","last_undegraded":"2024-04-25T01:25:17.955773+0000","last_fullsized":"2024-04-25T01:25:17.955773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:52.224547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956512+0000","last_change":"2024-04-25T01:25:16.737313+0000","last_active":"2024-04-25T01:25:17.956512+0000","last_peered":"2024-04-25T01:25:17.956512+0000","last_clean":"2024-04-25T01:25:17.956512+0000","last_became_active":"2024-04-25T01:25:16.736955+0000","last_became_peered":"2024-04-25T01:25:16.736955+0000","last_unstale":"2024-04-25T01:25:17.956512+0000","last_undegraded":"2024-04-25T01:25:17.956512+0000","last_fullsized":"2024-04-25T01:25:17.956512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:17.692960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955887+0000","last_change":"2024-04-25T01:25:16.750026+0000","last_active":"2024-04-25T01:25:17.955887+0000","last_peered":"2024-04-25T01:25:17.955887+0000","last_clean":"2024-04-25T01:25:17.955887+0000","last_became_active":"2024-04-25T01:25:16.749588+0000","last_became_peered":"2024-04-25T01:25:16.749588+0000","last_unstale":"2024-04-25T01:25:17.955887+0000","last_undegraded":"2024-04-25T01:25:17.955887+0000","last_fullsized":"2024-04-25T01:25:17.955887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:09.357719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956999+0000","last_change":"2024-04-25T01:25:16.743440+0000","last_active":"2024-04-25T01:25:17.956999+0000","last_peered":"2024-04-25T01:25:17.956999+0000","last_clean":"2024-04-25T01:25:17.956999+0000","last_became_active":"2024-04-25T01:25:16.742904+0000","last_became_peered":"2024-04-25T01:25:16.742904+0000","last_unstale":"2024-04-25T01:25:17.956999+0000","last_undegraded":"2024-04-25T01:25:17.956999+0000","last_fullsized":"2024-04-25T01:25:17.956999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:17.380260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731365+0000","last_change":"2024-04-25T01:25:16.738305+0000","last_active":"2024-04-25T01:25:17.731365+0000","last_peered":"2024-04-25T01:25:17.731365+0000","last_clean":"2024-04-25T01:25:17.731365+0000","last_became_active":"2024-04-25T01:25:16.737907+0000","last_became_peered":"2024-04-25T01:25:16.737907+0000","last_unstale":"2024-04-25T01:25:17.731365+0000","last_undegraded":"2024-04-25T01:25:17.731365+0000","last_fullsized":"2024-04-25T01:25:17.731365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:29.501388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956408+0000","last_change":"2024-04-25T01:25:16.757755+0000","last_active":"2024-04-25T01:25:17.956408+0000","last_peered":"2024-04-25T01:25:17.956408+0000","last_clean":"2024-04-25T01:25:17.956408+0000","last_became_active":"2024-04-25T01:25:16.757374+0000","last_became_peered":"2024-04-25T01:25:16.757374+0000","last_unstale":"2024-04-25T01:25:17.956408+0000","last_undegraded":"2024-04-25T01:25:17.956408+0000","last_fullsized":"2024-04-25T01:25:17.956408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:46.848407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250979,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":15,"num_read_kb":12,"num_write":113,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":88563,"data_compressed":3712,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1031457,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1031072,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":476714,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":460766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":438922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1030482,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":38193,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39543,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:25:23.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:25:23.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.002+0000 7f4cdec76640 1 -- 172.21.15.16:0/3316780259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd8103360 msgr2=0x7f4cd8109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.002+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3316780259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd8103360 0x7f4cd8109bf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009ca0 tx=0x7f4cd002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- 172.21.15.16:0/3316780259 shutdown_connections 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3316780259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd8103360 0x7f4cd8109bf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3316780259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd81029a0 0x7f4cd8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3316780259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd81017b0 0x7f4cd8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- 172.21.15.16:0/3316780259 >> 172.21.15.16:0/3316780259 conn(0x7f4cd80fcd40 msgr2=0x7f4cd80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- 172.21.15.16:0/3316780259 shutdown_connections 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- 172.21.15.16:0/3316780259 wait complete. 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 Processor -- start 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- start start 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd81017b0 0x7f4cd8197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 0x7f4cd8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8103360 0x7f4cd819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd810d2e0 con 0x7f4cd81029a0 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd810d160 con 0x7f4cd81017b0 2024-04-25T01:25:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cdec76640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd810d460 con 0x7f4cd8103360 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 0x7f4cd8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 0x7f4cd8197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32940/0 (socket says 172.21.15.16:32940) 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.003+0000 7f4cd7fff640 1 -- 172.21.15.16:0/3489907127 learned_addr learned my addr 172.21.15.16:0/3489907127 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cdca12640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd81017b0 0x7f4cd8197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cdd213640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8103360 0x7f4cd819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd7fff640 1 -- 172.21.15.16:0/3489907127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8103360 msgr2=0x7f4cd819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd7fff640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8103360 0x7f4cd819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd7fff640 1 -- 172.21.15.16:0/3489907127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd81017b0 msgr2=0x7f4cd8197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd7fff640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd81017b0 0x7f4cd8197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd7fff640 1 -- 172.21.15.16:0/3489907127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd819f5b0 con 0x7f4cd81029a0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd7fff640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 0x7f4cd8197ed0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400d950 tx=0x7f4cc400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4017030 con 0x7f4cd81029a0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd819f8a0 con 0x7f4cd81029a0 2024-04-25T01:25:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc4004020 con 0x7f4cd81029a0 2024-04-25T01:25:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40047a0 con 0x7f4cd81029a0 2024-04-25T01:25:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.004+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd819fe40 con 0x7f4cd81029a0 2024-04-25T01:25:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.005+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd8108b40 con 0x7f4cd81029a0 2024-04-25T01:25:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.006+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4cc40041d0 con 0x7f4cd81029a0 2024-04-25T01:25:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.007+0000 7f4cd5ffb640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc80a2c20 0x7f4cc80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.007+0000 7f4cdca12640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc80a2c20 0x7f4cc80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.007+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4cc40cec60 con 0x7f4cd81029a0 2024-04-25T01:25:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.007+0000 7f4cdca12640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc80a2c20 0x7f4cc80a50e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc005fd0 tx=0x7f4ccc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.011+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4cc409bb10 con 0x7f4cd81029a0 2024-04-25T01:25:24.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:24 smithi016 ceph-mon[29153]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:24.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:24 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2782064559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:24.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:24 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3163627776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:24.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.194+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4cd8061b80 con 0x7f4cc80a2c20 2024-04-25T01:25:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.202+0000 7f4cd5ffb640 1 -- 172.21.15.16:0/3489907127 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258894 (secure 0 0 0) 0x7f4cd8061b80 con 0x7f4cc80a2c20 2024-04-25T01:25:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:24.185 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc80a2c20 msgr2=0x7f4cc80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc80a2c20 0x7f4cc80a50e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc005fd0 tx=0x7f4ccc005850 comp rx=0 tx=0).stop 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 msgr2=0x7f4cd8197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 0x7f4cd8197ed0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400d950 tx=0x7f4cc400de10 comp rx=0 tx=0).stop 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 shutdown_connections 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc80a2c20 0x7f4cc80a50e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8103360 0x7f4cd819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81029a0 0x7f4cd8197ed0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 --2- 172.21.15.16:0/3489907127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd81017b0 0x7f4cd8197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 >> 172.21.15.16:0/3489907127 conn(0x7f4cd80fcd40 msgr2=0x7f4cd8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 shutdown_connections 2024-04-25T01:25:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:24.205+0000 7f4cdec76640 1 -- 172.21.15.16:0/3489907127 wait complete. 2024-04-25T01:25:24.196 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-04-25T01:25:23.610350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594968,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":141,"num_read_kb":121,"num_write":294,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368620,"kb_used_data":11112,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573204,"statfs":{"total":1151940427776,"available":1151562960896,"internally_reserved":0,"allocated":11378688,"data_stored":7104031,"data_compressed":111527,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":15,"num_read_kb":12,"num_write":113,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.647625"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957164+0000","last_change":"2024-04-25T01:25:16.740766+0000","last_active":"2024-04-25T01:25:17.957164+0000","last_peered":"2024-04-25T01:25:17.957164+0000","last_clean":"2024-04-25T01:25:17.957164+0000","last_became_active":"2024-04-25T01:25:16.740181+0000","last_became_peered":"2024-04-25T01:25:16.740181+0000","last_unstale":"2024-04-25T01:25:17.957164+0000","last_undegraded":"2024-04-25T01:25:17.957164+0000","last_fullsized":"2024-04-25T01:25:17.957164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:59.154389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955972+0000","last_change":"2024-04-25T01:25:16.750724+0000","last_active":"2024-04-25T01:25:17.955972+0000","last_peered":"2024-04-25T01:25:17.955972+0000","last_clean":"2024-04-25T01:25:17.955972+0000","last_became_active":"2024-04-25T01:25:16.750391+0000","last_became_peered":"2024-04-25T01:25:16.750391+0000","last_unstale":"2024-04-25T01:25:17.955972+0000","last_undegraded":"2024-04-25T01:25:17.955972+0000","last_fullsized":"2024-04-25T01:25:17.955972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:15.576020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956219+0000","last_change":"2024-04-25T01:25:16.753914+0000","last_active":"2024-04-25T01:25:17.956219+0000","last_peered":"2024-04-25T01:25:17.956219+0000","last_clean":"2024-04-25T01:25:17.956219+0000","last_became_active":"2024-04-25T01:25:16.753464+0000","last_became_peered":"2024-04-25T01:25:16.753464+0000","last_unstale":"2024-04-25T01:25:17.956219+0000","last_undegraded":"2024-04-25T01:25:17.956219+0000","last_fullsized":"2024-04-25T01:25:17.956219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:55.716948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957081+0000","last_change":"2024-04-25T01:25:16.754586+0000","last_active":"2024-04-25T01:25:17.957081+0000","last_peered":"2024-04-25T01:25:17.957081+0000","last_clean":"2024-04-25T01:25:17.957081+0000","last_became_active":"2024-04-25T01:25:16.752260+0000","last_became_peered":"2024-04-25T01:25:16.752260+0000","last_unstale":"2024-04-25T01:25:17.957081+0000","last_undegraded":"2024-04-25T01:25:17.957081+0000","last_fullsized":"2024-04-25T01:25:17.957081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:30.935184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956682+0000","last_change":"2024-04-25T01:25:16.741397+0000","last_active":"2024-04-25T01:25:17.956682+0000","last_peered":"2024-04-25T01:25:17.956682+0000","last_clean":"2024-04-25T01:25:17.956682+0000","last_became_active":"2024-04-25T01:25:16.741056+0000","last_became_peered":"2024-04-25T01:25:16.741056+0000","last_unstale":"2024-04-25T01:25:17.956682+0000","last_undegraded":"2024-04-25T01:25:17.956682+0000","last_fullsized":"2024-04-25T01:25:17.956682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:38.886023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737037+0000","last_change":"2024-04-25T01:25:16.757351+0000","last_active":"2024-04-25T01:25:17.737037+0000","last_peered":"2024-04-25T01:25:17.737037+0000","last_clean":"2024-04-25T01:25:17.737037+0000","last_became_active":"2024-04-25T01:25:16.756980+0000","last_became_peered":"2024-04-25T01:25:16.756980+0000","last_unstale":"2024-04-25T01:25:17.737037+0000","last_undegraded":"2024-04-25T01:25:17.737037+0000","last_fullsized":"2024-04-25T01:25:17.737037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:18.755302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541243+0000","last_change":"2024-04-25T01:25:16.740265+0000","last_active":"2024-04-25T01:25:18.541243+0000","last_peered":"2024-04-25T01:25:18.541243+0000","last_clean":"2024-04-25T01:25:18.541243+0000","last_became_active":"2024-04-25T01:25:16.739345+0000","last_became_peered":"2024-04-25T01:25:16.739345+0000","last_unstale":"2024-04-25T01:25:18.541243+0000","last_undegraded":"2024-04-25T01:25:18.541243+0000","last_fullsized":"2024-04-25T01:25:18.541243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:55.399650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955587+0000","last_change":"2024-04-25T01:25:16.753159+0000","last_active":"2024-04-25T01:25:17.955587+0000","last_peered":"2024-04-25T01:25:17.955587+0000","last_clean":"2024-04-25T01:25:17.955587+0000","last_became_active":"2024-04-25T01:25:16.751863+0000","last_became_peered":"2024-04-25T01:25:16.751863+0000","last_unstale":"2024-04-25T01:25:17.955587+0000","last_undegraded":"2024-04-25T01:25:17.955587+0000","last_fullsized":"2024-04-25T01:25:17.955587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:54.218096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957134+0000","last_change":"2024-04-25T01:25:16.738727+0000","last_active":"2024-04-25T01:25:17.957134+0000","last_peered":"2024-04-25T01:25:17.957134+0000","last_clean":"2024-04-25T01:25:17.957134+0000","last_became_active":"2024-04-25T01:25:16.738390+0000","last_became_peered":"2024-04-25T01:25:16.738390+0000","last_unstale":"2024-04-25T01:25:17.957134+0000","last_undegraded":"2024-04-25T01:25:17.957134+0000","last_fullsized":"2024-04-25T01:25:17.957134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:03.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737383+0000","last_change":"2024-04-25T01:25:16.756974+0000","last_active":"2024-04-25T01:25:17.737383+0000","last_peered":"2024-04-25T01:25:17.737383+0000","last_clean":"2024-04-25T01:25:17.737383+0000","last_became_active":"2024-04-25T01:25:16.756549+0000","last_became_peered":"2024-04-25T01:25:16.756549+0000","last_unstale":"2024-04-25T01:25:17.737383+0000","last_undegraded":"2024-04-25T01:25:17.737383+0000","last_fullsized":"2024-04-25T01:25:17.737383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:49.042132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541532+0000","last_change":"2024-04-25T01:25:16.745728+0000","last_active":"2024-04-25T01:25:18.541532+0000","last_peered":"2024-04-25T01:25:18.541532+0000","last_clean":"2024-04-25T01:25:18.541532+0000","last_became_active":"2024-04-25T01:25:16.745091+0000","last_became_peered":"2024-04-25T01:25:16.745091+0000","last_unstale":"2024-04-25T01:25:18.541532+0000","last_undegraded":"2024-04-25T01:25:18.541532+0000","last_fullsized":"2024-04-25T01:25:18.541532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:01.622335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957017+0000","last_change":"2024-04-25T01:25:16.755146+0000","last_active":"2024-04-25T01:25:17.957017+0000","last_peered":"2024-04-25T01:25:17.957017+0000","last_clean":"2024-04-25T01:25:17.957017+0000","last_became_active":"2024-04-25T01:25:16.752713+0000","last_became_peered":"2024-04-25T01:25:16.752713+0000","last_unstale":"2024-04-25T01:25:17.957017+0000","last_undegraded":"2024-04-25T01:25:17.957017+0000","last_fullsized":"2024-04-25T01:25:17.957017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:31.967903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731442+0000","last_change":"2024-04-25T01:25:16.751162+0000","last_active":"2024-04-25T01:25:17.731442+0000","last_peered":"2024-04-25T01:25:17.731442+0000","last_clean":"2024-04-25T01:25:17.731442+0000","last_became_active":"2024-04-25T01:25:16.750772+0000","last_became_peered":"2024-04-25T01:25:16.750772+0000","last_unstale":"2024-04-25T01:25:17.731442+0000","last_undegraded":"2024-04-25T01:25:17.731442+0000","last_fullsized":"2024-04-25T01:25:17.731442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:39.533120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955886+0000","last_change":"2024-04-25T01:25:16.754367+0000","last_active":"2024-04-25T01:25:17.955886+0000","last_peered":"2024-04-25T01:25:17.955886+0000","last_clean":"2024-04-25T01:25:17.955886+0000","last_became_active":"2024-04-25T01:25:16.750324+0000","last_became_peered":"2024-04-25T01:25:16.750324+0000","last_unstale":"2024-04-25T01:25:17.955886+0000","last_undegraded":"2024-04-25T01:25:17.955886+0000","last_fullsized":"2024-04-25T01:25:17.955886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:17.606187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955617+0000","last_change":"2024-04-25T01:25:16.753111+0000","last_active":"2024-04-25T01:25:17.955617+0000","last_peered":"2024-04-25T01:25:17.955617+0000","last_clean":"2024-04-25T01:25:17.955617+0000","last_became_active":"2024-04-25T01:25:16.752749+0000","last_became_peered":"2024-04-25T01:25:16.752749+0000","last_unstale":"2024-04-25T01:25:17.955617+0000","last_undegraded":"2024-04-25T01:25:17.955617+0000","last_fullsized":"2024-04-25T01:25:17.955617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:07:28.653568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956504+0000","last_change":"2024-04-25T01:25:16.744958+0000","last_active":"2024-04-25T01:25:17.956504+0000","last_peered":"2024-04-25T01:25:17.956504+0000","last_clean":"2024-04-25T01:25:17.956504+0000","last_became_active":"2024-04-25T01:25:16.744606+0000","last_became_peered":"2024-04-25T01:25:16.744606+0000","last_unstale":"2024-04-25T01:25:17.956504+0000","last_undegraded":"2024-04-25T01:25:17.956504+0000","last_fullsized":"2024-04-25T01:25:17.956504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:07:58.920942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957454+0000","last_change":"2024-04-25T01:25:16.743462+0000","last_active":"2024-04-25T01:25:17.957454+0000","last_peered":"2024-04-25T01:25:17.957454+0000","last_clean":"2024-04-25T01:25:17.957454+0000","last_became_active":"2024-04-25T01:25:16.743055+0000","last_became_peered":"2024-04-25T01:25:16.743055+0000","last_unstale":"2024-04-25T01:25:17.957454+0000","last_undegraded":"2024-04-25T01:25:17.957454+0000","last_fullsized":"2024-04-25T01:25:17.957454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:23:03.660065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956737+0000","last_change":"2024-04-25T01:25:16.751446+0000","last_active":"2024-04-25T01:25:17.956737+0000","last_peered":"2024-04-25T01:25:17.956737+0000","last_clean":"2024-04-25T01:25:17.956737+0000","last_became_active":"2024-04-25T01:25:16.751107+0000","last_became_peered":"2024-04-25T01:25:16.751107+0000","last_unstale":"2024-04-25T01:25:17.956737+0000","last_undegraded":"2024-04-25T01:25:17.956737+0000","last_fullsized":"2024-04-25T01:25:17.956737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:52.275662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.732141+0000","last_change":"2024-04-25T01:25:16.748478+0000","last_active":"2024-04-25T01:25:17.732141+0000","last_peered":"2024-04-25T01:25:17.732141+0000","last_clean":"2024-04-25T01:25:17.732141+0000","last_became_active":"2024-04-25T01:25:16.748064+0000","last_became_peered":"2024-04-25T01:25:16.748064+0000","last_unstale":"2024-04-25T01:25:17.732141+0000","last_undegraded":"2024-04-25T01:25:17.732141+0000","last_fullsized":"2024-04-25T01:25:17.732141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:45.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956794+0000","last_change":"2024-04-25T01:25:15.987611+0000","last_active":"2024-04-25T01:25:17.956794+0000","last_peered":"2024-04-25T01:25:17.956794+0000","last_clean":"2024-04-25T01:25:17.956794+0000","last_became_active":"2024-04-25T01:25:15.987300+0000","last_became_peered":"2024-04-25T01:25:15.987300+0000","last_unstale":"2024-04-25T01:25:17.956794+0000","last_undegraded":"2024-04-25T01:25:17.956794+0000","last_fullsized":"2024-04-25T01:25:17.956794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:24.636607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955696+0000","last_change":"2024-04-25T01:25:16.751076+0000","last_active":"2024-04-25T01:25:17.955696+0000","last_peered":"2024-04-25T01:25:17.955696+0000","last_clean":"2024-04-25T01:25:17.955696+0000","last_became_active":"2024-04-25T01:25:16.750219+0000","last_became_peered":"2024-04-25T01:25:16.750219+0000","last_unstale":"2024-04-25T01:25:17.955696+0000","last_undegraded":"2024-04-25T01:25:17.955696+0000","last_fullsized":"2024-04-25T01:25:17.955696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:11.956004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956263+0000","last_change":"2024-04-25T01:25:15.977434+0000","last_active":"2024-04-25T01:25:17.956263+0000","last_peered":"2024-04-25T01:25:17.956263+0000","last_clean":"2024-04-25T01:25:17.956263+0000","last_became_active":"2024-04-25T01:25:15.977095+0000","last_became_peered":"2024-04-25T01:25:15.977095+0000","last_unstale":"2024-04-25T01:25:17.956263+0000","last_undegraded":"2024-04-25T01:25:17.956263+0000","last_fullsized":"2024-04-25T01:25:17.956263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:01.102962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956536+0000","last_change":"2024-04-25T01:25:16.740061+0000","last_active":"2024-04-25T01:25:17.956536+0000","last_peered":"2024-04-25T01:25:17.956536+0000","last_clean":"2024-04-25T01:25:17.956536+0000","last_became_active":"2024-04-25T01:25:16.739593+0000","last_became_peered":"2024-04-25T01:25:16.739593+0000","last_unstale":"2024-04-25T01:25:17.956536+0000","last_undegraded":"2024-04-25T01:25:17.956536+0000","last_fullsized":"2024-04-25T01:25:17.956536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:32.711066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.071581+0000","last_change":"2024-04-25T01:25:15.984150+0000","last_active":"2024-04-25T01:25:20.071581+0000","last_peered":"2024-04-25T01:25:20.071581+0000","last_clean":"2024-04-25T01:25:20.071581+0000","last_became_active":"2024-04-25T01:25:15.983405+0000","last_became_peered":"2024-04-25T01:25:15.983405+0000","last_unstale":"2024-04-25T01:25:20.071581+0000","last_undegraded":"2024-04-25T01:25:20.071581+0000","last_fullsized":"2024-04-25T01:25:20.071581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:38.511851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956437+0000","last_change":"2024-04-25T01:25:16.751549+0000","last_active":"2024-04-25T01:25:17.956437+0000","last_peered":"2024-04-25T01:25:17.956437+0000","last_clean":"2024-04-25T01:25:17.956437+0000","last_became_active":"2024-04-25T01:25:16.751167+0000","last_became_peered":"2024-04-25T01:25:16.751167+0000","last_unstale":"2024-04-25T01:25:17.956437+0000","last_undegraded":"2024-04-25T01:25:17.956437+0000","last_fullsized":"2024-04-25T01:25:17.956437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:21.260019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957152+0000","last_change":"2024-04-25T01:25:15.977916+0000","last_active":"2024-04-25T01:25:17.957152+0000","last_peered":"2024-04-25T01:25:17.957152+0000","last_clean":"2024-04-25T01:25:17.957152+0000","last_became_active":"2024-04-25T01:25:15.977602+0000","last_became_peered":"2024-04-25T01:25:15.977602+0000","last_unstale":"2024-04-25T01:25:17.957152+0000","last_undegraded":"2024-04-25T01:25:17.957152+0000","last_fullsized":"2024-04-25T01:25:17.957152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:27.926489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957490+0000","last_change":"2024-04-25T01:25:16.756728+0000","last_active":"2024-04-25T01:25:17.957490+0000","last_peered":"2024-04-25T01:25:17.957490+0000","last_clean":"2024-04-25T01:25:17.957490+0000","last_became_active":"2024-04-25T01:25:16.756363+0000","last_became_peered":"2024-04-25T01:25:16.756363+0000","last_unstale":"2024-04-25T01:25:17.957490+0000","last_undegraded":"2024-04-25T01:25:17.957490+0000","last_fullsized":"2024-04-25T01:25:17.957490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:06.582831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:22.079357+0000","last_change":"2024-04-25T01:25:15.987099+0000","last_active":"2024-04-25T01:25:22.079357+0000","last_peered":"2024-04-25T01:25:22.079357+0000","last_clean":"2024-04-25T01:25:22.079357+0000","last_became_active":"2024-04-25T01:25:15.986745+0000","last_became_peered":"2024-04-25T01:25:15.986745+0000","last_unstale":"2024-04-25T01:25:22.079357+0000","last_undegraded":"2024-04-25T01:25:22.079357+0000","last_fullsized":"2024-04-25T01:25:22.079357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:14.944342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956643+0000","last_change":"2024-04-25T01:25:16.742735+0000","last_active":"2024-04-25T01:25:17.956643+0000","last_peered":"2024-04-25T01:25:17.956643+0000","last_clean":"2024-04-25T01:25:17.956643+0000","last_became_active":"2024-04-25T01:25:16.742391+0000","last_became_peered":"2024-04-25T01:25:16.742391+0000","last_unstale":"2024-04-25T01:25:17.956643+0000","last_undegraded":"2024-04-25T01:25:17.956643+0000","last_fullsized":"2024-04-25T01:25:17.956643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:05.783111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073249+0000","last_change":"2024-04-25T01:25:15.982605+0000","last_active":"2024-04-25T01:25:20.073249+0000","last_peered":"2024-04-25T01:25:20.073249+0000","last_clean":"2024-04-25T01:25:20.073249+0000","last_became_active":"2024-04-25T01:25:15.982264+0000","last_became_peered":"2024-04-25T01:25:15.982264+0000","last_unstale":"2024-04-25T01:25:20.073249+0000","last_undegraded":"2024-04-25T01:25:20.073249+0000","last_fullsized":"2024-04-25T01:25:20.073249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:09.960827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957619+0000","last_change":"2024-04-25T01:25:16.747784+0000","last_active":"2024-04-25T01:25:17.957619+0000","last_peered":"2024-04-25T01:25:17.957619+0000","last_clean":"2024-04-25T01:25:17.957619+0000","last_became_active":"2024-04-25T01:25:16.747432+0000","last_became_peered":"2024-04-25T01:25:16.747432+0000","last_unstale":"2024-04-25T01:25:17.957619+0000","last_undegraded":"2024-04-25T01:25:17.957619+0000","last_fullsized":"2024-04-25T01:25:17.957619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:40.602582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.074441+0000","last_change":"2024-04-25T01:25:15.977370+0000","last_active":"2024-04-25T01:25:20.074441+0000","last_peered":"2024-04-25T01:25:20.074441+0000","last_clean":"2024-04-25T01:25:20.074441+0000","last_became_active":"2024-04-25T01:25:15.977010+0000","last_became_peered":"2024-04-25T01:25:15.977010+0000","last_unstale":"2024-04-25T01:25:20.074441+0000","last_undegraded":"2024-04-25T01:25:20.074441+0000","last_fullsized":"2024-04-25T01:25:20.074441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:41.239154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955636+0000","last_change":"2024-04-25T01:25:16.741490+0000","last_active":"2024-04-25T01:25:17.955636+0000","last_peered":"2024-04-25T01:25:17.955636+0000","last_clean":"2024-04-25T01:25:17.955636+0000","last_became_active":"2024-04-25T01:25:16.741148+0000","last_became_peered":"2024-04-25T01:25:16.741148+0000","last_unstale":"2024-04-25T01:25:17.955636+0000","last_undegraded":"2024-04-25T01:25:17.955636+0000","last_fullsized":"2024-04-25T01:25:17.955636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:50.918575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.440350+0000","last_change":"2024-04-25T01:25:15.988283+0000","last_active":"2024-04-25T01:25:18.440350+0000","last_peered":"2024-04-25T01:25:18.440350+0000","last_clean":"2024-04-25T01:25:18.440350+0000","last_became_active":"2024-04-25T01:25:15.987894+0000","last_became_peered":"2024-04-25T01:25:15.987894+0000","last_unstale":"2024-04-25T01:25:18.440350+0000","last_undegraded":"2024-04-25T01:25:18.440350+0000","last_fullsized":"2024-04-25T01:25:18.440350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:01.220170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956377+0000","last_change":"2024-04-25T01:25:16.749468+0000","last_active":"2024-04-25T01:25:17.956377+0000","last_peered":"2024-04-25T01:25:17.956377+0000","last_clean":"2024-04-25T01:25:17.956377+0000","last_became_active":"2024-04-25T01:25:16.749094+0000","last_became_peered":"2024-04-25T01:25:16.749094+0000","last_unstale":"2024-04-25T01:25:17.956377+0000","last_undegraded":"2024-04-25T01:25:17.956377+0000","last_fullsized":"2024-04-25T01:25:17.956377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:17.701149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.067512+0000","last_change":"2024-04-25T01:25:15.981590+0000","last_active":"2024-04-25T01:25:19.067512+0000","last_peered":"2024-04-25T01:25:19.067512+0000","last_clean":"2024-04-25T01:25:19.067512+0000","last_became_active":"2024-04-25T01:25:15.981192+0000","last_became_peered":"2024-04-25T01:25:15.981192+0000","last_unstale":"2024-04-25T01:25:19.067512+0000","last_undegraded":"2024-04-25T01:25:19.067512+0000","last_fullsized":"2024-04-25T01:25:19.067512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:06.220365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956930+0000","last_change":"2024-04-25T01:25:16.751414+0000","last_active":"2024-04-25T01:25:17.956930+0000","last_peered":"2024-04-25T01:25:17.956930+0000","last_clean":"2024-04-25T01:25:17.956930+0000","last_became_active":"2024-04-25T01:25:16.751003+0000","last_became_peered":"2024-04-25T01:25:16.751003+0000","last_unstale":"2024-04-25T01:25:17.956930+0000","last_undegraded":"2024-04-25T01:25:17.956930+0000","last_fullsized":"2024-04-25T01:25:17.956930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:49:03.146685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068928+0000","last_change":"2024-04-25T01:25:15.974041+0000","last_active":"2024-04-25T01:25:19.068928+0000","last_peered":"2024-04-25T01:25:19.068928+0000","last_clean":"2024-04-25T01:25:19.068928+0000","last_became_active":"2024-04-25T01:25:15.973621+0000","last_became_peered":"2024-04-25T01:25:15.973621+0000","last_unstale":"2024-04-25T01:25:19.068928+0000","last_undegraded":"2024-04-25T01:25:19.068928+0000","last_fullsized":"2024-04-25T01:25:19.068928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:12.039161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956496+0000","last_change":"2024-04-25T01:25:16.745295+0000","last_active":"2024-04-25T01:25:17.956496+0000","last_peered":"2024-04-25T01:25:17.956496+0000","last_clean":"2024-04-25T01:25:17.956496+0000","last_became_active":"2024-04-25T01:25:16.744931+0000","last_became_peered":"2024-04-25T01:25:16.744931+0000","last_unstale":"2024-04-25T01:25:17.956496+0000","last_undegraded":"2024-04-25T01:25:17.956496+0000","last_fullsized":"2024-04-25T01:25:17.956496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:48.993537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.067512+0000","last_change":"2024-04-25T01:25:15.980900+0000","last_active":"2024-04-25T01:25:19.067512+0000","last_peered":"2024-04-25T01:25:19.067512+0000","last_clean":"2024-04-25T01:25:19.067512+0000","last_became_active":"2024-04-25T01:25:15.980528+0000","last_became_peered":"2024-04-25T01:25:15.980528+0000","last_unstale":"2024-04-25T01:25:19.067512+0000","last_undegraded":"2024-04-25T01:25:19.067512+0000","last_fullsized":"2024-04-25T01:25:19.067512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:47.743151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956549+0000","last_change":"2024-04-25T01:25:16.753886+0000","last_active":"2024-04-25T01:25:17.956549+0000","last_peered":"2024-04-25T01:25:17.956549+0000","last_clean":"2024-04-25T01:25:17.956549+0000","last_became_active":"2024-04-25T01:25:16.753522+0000","last_became_peered":"2024-04-25T01:25:16.753522+0000","last_unstale":"2024-04-25T01:25:17.956549+0000","last_undegraded":"2024-04-25T01:25:17.956549+0000","last_fullsized":"2024-04-25T01:25:17.956549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:12:46.103249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.065934+0000","last_change":"2024-04-25T01:25:15.980114+0000","last_active":"2024-04-25T01:25:19.065934+0000","last_peered":"2024-04-25T01:25:19.065934+0000","last_clean":"2024-04-25T01:25:19.065934+0000","last_became_active":"2024-04-25T01:25:15.979676+0000","last_became_peered":"2024-04-25T01:25:15.979676+0000","last_unstale":"2024-04-25T01:25:19.065934+0000","last_undegraded":"2024-04-25T01:25:19.065934+0000","last_fullsized":"2024-04-25T01:25:19.065934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:07.401375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956419+0000","last_change":"2024-04-25T01:25:16.739726+0000","last_active":"2024-04-25T01:25:17.956419+0000","last_peered":"2024-04-25T01:25:17.956419+0000","last_clean":"2024-04-25T01:25:17.956419+0000","last_became_active":"2024-04-25T01:25:16.739399+0000","last_became_peered":"2024-04-25T01:25:16.739399+0000","last_unstale":"2024-04-25T01:25:17.956419+0000","last_undegraded":"2024-04-25T01:25:17.956419+0000","last_fullsized":"2024-04-25T01:25:17.956419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:21:42.747061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.072422+0000","last_change":"2024-04-25T01:25:15.976731+0000","last_active":"2024-04-25T01:25:20.072422+0000","last_peered":"2024-04-25T01:25:20.072422+0000","last_clean":"2024-04-25T01:25:20.072422+0000","last_became_active":"2024-04-25T01:25:15.976405+0000","last_became_peered":"2024-04-25T01:25:15.976405+0000","last_unstale":"2024-04-25T01:25:20.072422+0000","last_undegraded":"2024-04-25T01:25:20.072422+0000","last_fullsized":"2024-04-25T01:25:20.072422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:21.612389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731512+0000","last_change":"2024-04-25T01:25:16.741378+0000","last_active":"2024-04-25T01:25:17.731512+0000","last_peered":"2024-04-25T01:25:17.731512+0000","last_clean":"2024-04-25T01:25:17.731512+0000","last_became_active":"2024-04-25T01:25:16.741066+0000","last_became_peered":"2024-04-25T01:25:16.741066+0000","last_unstale":"2024-04-25T01:25:17.731512+0000","last_undegraded":"2024-04-25T01:25:17.731512+0000","last_fullsized":"2024-04-25T01:25:17.731512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:37.288181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068861+0000","last_change":"2024-04-25T01:25:15.988288+0000","last_active":"2024-04-25T01:25:19.068861+0000","last_peered":"2024-04-25T01:25:19.068861+0000","last_clean":"2024-04-25T01:25:19.068861+0000","last_became_active":"2024-04-25T01:25:15.987860+0000","last_became_peered":"2024-04-25T01:25:15.987860+0000","last_unstale":"2024-04-25T01:25:19.068861+0000","last_undegraded":"2024-04-25T01:25:19.068861+0000","last_fullsized":"2024-04-25T01:25:19.068861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:23.634199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541710+0000","last_change":"2024-04-25T01:25:16.743784+0000","last_active":"2024-04-25T01:25:18.541710+0000","last_peered":"2024-04-25T01:25:18.541710+0000","last_clean":"2024-04-25T01:25:18.541710+0000","last_became_active":"2024-04-25T01:25:16.743435+0000","last_became_peered":"2024-04-25T01:25:16.743435+0000","last_unstale":"2024-04-25T01:25:18.541710+0000","last_undegraded":"2024-04-25T01:25:18.541710+0000","last_fullsized":"2024-04-25T01:25:18.541710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:52:08.297617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956027+0000","last_change":"2024-04-25T01:25:15.978201+0000","last_active":"2024-04-25T01:25:17.956027+0000","last_peered":"2024-04-25T01:25:17.956027+0000","last_clean":"2024-04-25T01:25:17.956027+0000","last_became_active":"2024-04-25T01:25:15.977832+0000","last_became_peered":"2024-04-25T01:25:15.977832+0000","last_unstale":"2024-04-25T01:25:17.956027+0000","last_undegraded":"2024-04-25T01:25:17.956027+0000","last_fullsized":"2024-04-25T01:25:17.956027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:42.483969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541659+0000","last_change":"2024-04-25T01:25:16.750944+0000","last_active":"2024-04-25T01:25:18.541659+0000","last_peered":"2024-04-25T01:25:18.541659+0000","last_clean":"2024-04-25T01:25:18.541659+0000","last_became_active":"2024-04-25T01:25:16.750588+0000","last_became_peered":"2024-04-25T01:25:16.750588+0000","last_unstale":"2024-04-25T01:25:18.541659+0000","last_undegraded":"2024-04-25T01:25:18.541659+0000","last_fullsized":"2024-04-25T01:25:18.541659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:31.473184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.075003+0000","last_change":"2024-04-25T01:25:15.976656+0000","last_active":"2024-04-25T01:25:20.075003+0000","last_peered":"2024-04-25T01:25:20.075003+0000","last_clean":"2024-04-25T01:25:20.075003+0000","last_became_active":"2024-04-25T01:25:15.976318+0000","last_became_peered":"2024-04-25T01:25:15.976318+0000","last_unstale":"2024-04-25T01:25:20.075003+0000","last_undegraded":"2024-04-25T01:25:20.075003+0000","last_fullsized":"2024-04-25T01:25:20.075003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:49.702258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956906+0000","last_change":"2024-04-25T01:25:16.748178+0000","last_active":"2024-04-25T01:25:17.956906+0000","last_peered":"2024-04-25T01:25:17.956906+0000","last_clean":"2024-04-25T01:25:17.956906+0000","last_became_active":"2024-04-25T01:25:16.747868+0000","last_became_peered":"2024-04-25T01:25:16.747868+0000","last_unstale":"2024-04-25T01:25:17.956906+0000","last_undegraded":"2024-04-25T01:25:17.956906+0000","last_fullsized":"2024-04-25T01:25:17.956906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:06.378387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:21.068756+0000","last_change":"2024-04-25T01:25:15.981651+0000","last_active":"2024-04-25T01:25:21.068756+0000","last_peered":"2024-04-25T01:25:21.068756+0000","last_clean":"2024-04-25T01:25:21.068756+0000","last_became_active":"2024-04-25T01:25:15.981245+0000","last_became_peered":"2024-04-25T01:25:15.981245+0000","last_unstale":"2024-04-25T01:25:21.068756+0000","last_undegraded":"2024-04-25T01:25:21.068756+0000","last_fullsized":"2024-04-25T01:25:21.068756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:20.009347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956478+0000","last_change":"2024-04-25T01:25:16.756873+0000","last_active":"2024-04-25T01:25:17.956478+0000","last_peered":"2024-04-25T01:25:17.956478+0000","last_clean":"2024-04-25T01:25:17.956478+0000","last_became_active":"2024-04-25T01:25:16.756505+0000","last_became_peered":"2024-04-25T01:25:16.756505+0000","last_unstale":"2024-04-25T01:25:17.956478+0000","last_undegraded":"2024-04-25T01:25:17.956478+0000","last_fullsized":"2024-04-25T01:25:17.956478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:23.412303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956708+0000","last_change":"2024-04-25T01:25:15.983640+0000","last_active":"2024-04-25T01:25:17.956708+0000","last_peered":"2024-04-25T01:25:17.956708+0000","last_clean":"2024-04-25T01:25:17.956708+0000","last_became_active":"2024-04-25T01:25:15.983280+0000","last_became_peered":"2024-04-25T01:25:15.983280+0000","last_unstale":"2024-04-25T01:25:17.956708+0000","last_undegraded":"2024-04-25T01:25:17.956708+0000","last_fullsized":"2024-04-25T01:25:17.956708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:18.493139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737350+0000","last_change":"2024-04-25T01:25:16.751616+0000","last_active":"2024-04-25T01:25:17.737350+0000","last_peered":"2024-04-25T01:25:17.737350+0000","last_clean":"2024-04-25T01:25:17.737350+0000","last_became_active":"2024-04-25T01:25:16.751254+0000","last_became_peered":"2024-04-25T01:25:16.751254+0000","last_unstale":"2024-04-25T01:25:17.737350+0000","last_undegraded":"2024-04-25T01:25:17.737350+0000","last_fullsized":"2024-04-25T01:25:17.737350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:55.556054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541568+0000","last_change":"2024-04-25T01:25:15.977774+0000","last_active":"2024-04-25T01:25:18.541568+0000","last_peered":"2024-04-25T01:25:18.541568+0000","last_clean":"2024-04-25T01:25:18.541568+0000","last_became_active":"2024-04-25T01:25:15.977398+0000","last_became_peered":"2024-04-25T01:25:15.977398+0000","last_unstale":"2024-04-25T01:25:18.541568+0000","last_undegraded":"2024-04-25T01:25:18.541568+0000","last_fullsized":"2024-04-25T01:25:18.541568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:33.729091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.737011+0000","last_change":"2024-04-25T01:25:16.749604+0000","last_active":"2024-04-25T01:25:17.737011+0000","last_peered":"2024-04-25T01:25:17.737011+0000","last_clean":"2024-04-25T01:25:17.737011+0000","last_became_active":"2024-04-25T01:25:16.749236+0000","last_became_peered":"2024-04-25T01:25:16.749236+0000","last_unstale":"2024-04-25T01:25:17.737011+0000","last_undegraded":"2024-04-25T01:25:17.737011+0000","last_fullsized":"2024-04-25T01:25:17.737011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:20:54.362813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.957283+0000","last_change":"2024-04-25T01:25:16.746447+0000","last_active":"2024-04-25T01:25:17.957283+0000","last_peered":"2024-04-25T01:25:17.957283+0000","last_clean":"2024-04-25T01:25:17.957283+0000","last_became_active":"2024-04-25T01:25:16.745978+0000","last_became_peered":"2024-04-25T01:25:16.745978+0000","last_unstale":"2024-04-25T01:25:17.957283+0000","last_undegraded":"2024-04-25T01:25:17.957283+0000","last_fullsized":"2024-04-25T01:25:17.957283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:41.249662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073887+0000","last_change":"2024-04-25T01:25:15.987083+0000","last_active":"2024-04-25T01:25:20.073887+0000","last_peered":"2024-04-25T01:25:20.073887+0000","last_clean":"2024-04-25T01:25:20.073887+0000","last_became_active":"2024-04-25T01:25:15.986632+0000","last_became_peered":"2024-04-25T01:25:15.986632+0000","last_unstale":"2024-04-25T01:25:20.073887+0000","last_undegraded":"2024-04-25T01:25:20.073887+0000","last_fullsized":"2024-04-25T01:25:20.073887+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:44.792239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956209+0000","last_change":"2024-04-25T01:25:16.742007+0000","last_active":"2024-04-25T01:25:17.956209+0000","last_peered":"2024-04-25T01:25:17.956209+0000","last_clean":"2024-04-25T01:25:17.956209+0000","last_became_active":"2024-04-25T01:25:16.741464+0000","last_became_peered":"2024-04-25T01:25:16.741464+0000","last_unstale":"2024-04-25T01:25:17.956209+0000","last_undegraded":"2024-04-25T01:25:17.956209+0000","last_fullsized":"2024-04-25T01:25:17.956209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:32.171648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068452+0000","last_change":"2024-04-25T01:25:15.978255+0000","last_active":"2024-04-25T01:25:19.068452+0000","last_peered":"2024-04-25T01:25:19.068452+0000","last_clean":"2024-04-25T01:25:19.068452+0000","last_became_active":"2024-04-25T01:25:15.977860+0000","last_became_peered":"2024-04-25T01:25:15.977860+0000","last_unstale":"2024-04-25T01:25:19.068452+0000","last_undegraded":"2024-04-25T01:25:19.068452+0000","last_fullsized":"2024-04-25T01:25:19.068452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.516137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":339,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541038+0000","last_change":"2024-04-25T01:23:58.204099+0000","last_active":"2024-04-25T01:25:18.541038+0000","last_peered":"2024-04-25T01:25:18.541038+0000","last_clean":"2024-04-25T01:25:18.541038+0000","last_became_active":"2024-04-25T01:23:58.203533+0000","last_became_peered":"2024-04-25T01:23:58.203533+0000","last_unstale":"2024-04-25T01:25:18.541038+0000","last_undegraded":"2024-04-25T01:25:18.541038+0000","last_fullsized":"2024-04-25T01:25:18.541038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_clean_scrub_stamp":"2024-04-25T01:20:28.584758+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:17.504360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731154+0000","last_change":"2024-04-25T01:25:16.746335+0000","last_active":"2024-04-25T01:25:17.731154+0000","last_peered":"2024-04-25T01:25:17.731154+0000","last_clean":"2024-04-25T01:25:17.731154+0000","last_became_active":"2024-04-25T01:25:16.745878+0000","last_became_peered":"2024-04-25T01:25:16.745878+0000","last_unstale":"2024-04-25T01:25:17.731154+0000","last_undegraded":"2024-04-25T01:25:17.731154+0000","last_fullsized":"2024-04-25T01:25:17.731154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:15:35.318542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073285+0000","last_change":"2024-04-25T01:25:15.986400+0000","last_active":"2024-04-25T01:25:20.073285+0000","last_peered":"2024-04-25T01:25:20.073285+0000","last_clean":"2024-04-25T01:25:20.073285+0000","last_became_active":"2024-04-25T01:25:15.985975+0000","last_became_peered":"2024-04-25T01:25:15.985975+0000","last_unstale":"2024-04-25T01:25:20.073285+0000","last_undegraded":"2024-04-25T01:25:20.073285+0000","last_fullsized":"2024-04-25T01:25:20.073285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:12.666394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.065873+0000","last_change":"2024-04-25T01:25:15.981370+0000","last_active":"2024-04-25T01:25:19.065873+0000","last_peered":"2024-04-25T01:25:19.065873+0000","last_clean":"2024-04-25T01:25:19.065873+0000","last_became_active":"2024-04-25T01:25:15.980995+0000","last_became_peered":"2024-04-25T01:25:15.980995+0000","last_unstale":"2024-04-25T01:25:19.065873+0000","last_undegraded":"2024-04-25T01:25:19.065873+0000","last_fullsized":"2024-04-25T01:25:19.065873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:29.754441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731403+0000","last_change":"2024-04-25T01:25:16.744887+0000","last_active":"2024-04-25T01:25:17.731403+0000","last_peered":"2024-04-25T01:25:17.731403+0000","last_clean":"2024-04-25T01:25:17.731403+0000","last_became_active":"2024-04-25T01:25:16.744538+0000","last_became_peered":"2024-04-25T01:25:16.744538+0000","last_unstale":"2024-04-25T01:25:17.731403+0000","last_undegraded":"2024-04-25T01:25:17.731403+0000","last_fullsized":"2024-04-25T01:25:17.731403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:32.132521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.048072+0000","last_change":"2024-04-25T01:25:15.990181+0000","last_active":"2024-04-25T01:25:19.048072+0000","last_peered":"2024-04-25T01:25:19.048072+0000","last_clean":"2024-04-25T01:25:19.048072+0000","last_became_active":"2024-04-25T01:25:15.989816+0000","last_became_peered":"2024-04-25T01:25:15.989816+0000","last_unstale":"2024-04-25T01:25:19.048072+0000","last_undegraded":"2024-04-25T01:25:19.048072+0000","last_fullsized":"2024-04-25T01:25:19.048072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:17:58.377057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956438+0000","last_change":"2024-04-25T01:25:16.746199+0000","last_active":"2024-04-25T01:25:17.956438+0000","last_peered":"2024-04-25T01:25:17.956438+0000","last_clean":"2024-04-25T01:25:17.956438+0000","last_became_active":"2024-04-25T01:25:16.745857+0000","last_became_peered":"2024-04-25T01:25:16.745857+0000","last_unstale":"2024-04-25T01:25:17.956438+0000","last_undegraded":"2024-04-25T01:25:17.956438+0000","last_fullsized":"2024-04-25T01:25:17.956438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:21.392309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068066+0000","last_change":"2024-04-25T01:25:15.981173+0000","last_active":"2024-04-25T01:25:19.068066+0000","last_peered":"2024-04-25T01:25:19.068066+0000","last_clean":"2024-04-25T01:25:19.068066+0000","last_became_active":"2024-04-25T01:25:15.980718+0000","last_became_peered":"2024-04-25T01:25:15.980718+0000","last_unstale":"2024-04-25T01:25:19.068066+0000","last_undegraded":"2024-04-25T01:25:19.068066+0000","last_fullsized":"2024-04-25T01:25:19.068066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:54.086543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956129+0000","last_change":"2024-04-25T01:25:16.753987+0000","last_active":"2024-04-25T01:25:17.956129+0000","last_peered":"2024-04-25T01:25:17.956129+0000","last_clean":"2024-04-25T01:25:17.956129+0000","last_became_active":"2024-04-25T01:25:16.753546+0000","last_became_peered":"2024-04-25T01:25:16.753546+0000","last_unstale":"2024-04-25T01:25:17.956129+0000","last_undegraded":"2024-04-25T01:25:17.956129+0000","last_fullsized":"2024-04-25T01:25:17.956129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:10.318501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.068770+0000","last_change":"2024-04-25T01:25:15.980938+0000","last_active":"2024-04-25T01:25:19.068770+0000","last_peered":"2024-04-25T01:25:19.068770+0000","last_clean":"2024-04-25T01:25:19.068770+0000","last_became_active":"2024-04-25T01:25:15.980586+0000","last_became_peered":"2024-04-25T01:25:15.980586+0000","last_unstale":"2024-04-25T01:25:19.068770+0000","last_undegraded":"2024-04-25T01:25:19.068770+0000","last_fullsized":"2024-04-25T01:25:19.068770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:00.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956369+0000","last_change":"2024-04-25T01:25:16.749711+0000","last_active":"2024-04-25T01:25:17.956369+0000","last_peered":"2024-04-25T01:25:17.956369+0000","last_clean":"2024-04-25T01:25:17.956369+0000","last_became_active":"2024-04-25T01:25:16.749218+0000","last_became_peered":"2024-04-25T01:25:16.749218+0000","last_unstale":"2024-04-25T01:25:17.956369+0000","last_undegraded":"2024-04-25T01:25:17.956369+0000","last_fullsized":"2024-04-25T01:25:17.956369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:56.555204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.067510+0000","last_change":"2024-04-25T01:25:15.988257+0000","last_active":"2024-04-25T01:25:19.067510+0000","last_peered":"2024-04-25T01:25:19.067510+0000","last_clean":"2024-04-25T01:25:19.067510+0000","last_became_active":"2024-04-25T01:25:15.987469+0000","last_became_peered":"2024-04-25T01:25:15.987469+0000","last_unstale":"2024-04-25T01:25:19.067510+0000","last_undegraded":"2024-04-25T01:25:19.067510+0000","last_fullsized":"2024-04-25T01:25:19.067510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:29.849868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956742+0000","last_change":"2024-04-25T01:25:16.746686+0000","last_active":"2024-04-25T01:25:17.956742+0000","last_peered":"2024-04-25T01:25:17.956742+0000","last_clean":"2024-04-25T01:25:17.956742+0000","last_became_active":"2024-04-25T01:25:16.746028+0000","last_became_peered":"2024-04-25T01:25:16.746028+0000","last_unstale":"2024-04-25T01:25:17.956742+0000","last_undegraded":"2024-04-25T01:25:17.956742+0000","last_fullsized":"2024-04-25T01:25:17.956742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:34:35.172207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731990+0000","last_change":"2024-04-25T01:25:15.974569+0000","last_active":"2024-04-25T01:25:17.731990+0000","last_peered":"2024-04-25T01:25:17.731990+0000","last_clean":"2024-04-25T01:25:17.731990+0000","last_became_active":"2024-04-25T01:25:15.973668+0000","last_became_peered":"2024-04-25T01:25:15.973668+0000","last_unstale":"2024-04-25T01:25:17.731990+0000","last_undegraded":"2024-04-25T01:25:17.731990+0000","last_fullsized":"2024-04-25T01:25:17.731990+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:02.813912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956677+0000","last_change":"2024-04-25T01:25:16.752510+0000","last_active":"2024-04-25T01:25:17.956677+0000","last_peered":"2024-04-25T01:25:17.956677+0000","last_clean":"2024-04-25T01:25:17.956677+0000","last_became_active":"2024-04-25T01:25:16.751889+0000","last_became_peered":"2024-04-25T01:25:16.751889+0000","last_unstale":"2024-04-25T01:25:17.956677+0000","last_undegraded":"2024-04-25T01:25:17.956677+0000","last_fullsized":"2024-04-25T01:25:17.956677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:44.628980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:19.069039+0000","last_change":"2024-04-25T01:25:15.983638+0000","last_active":"2024-04-25T01:25:19.069039+0000","last_peered":"2024-04-25T01:25:19.069039+0000","last_clean":"2024-04-25T01:25:19.069039+0000","last_became_active":"2024-04-25T01:25:15.983293+0000","last_became_peered":"2024-04-25T01:25:15.983293+0000","last_unstale":"2024-04-25T01:25:19.069039+0000","last_undegraded":"2024-04-25T01:25:19.069039+0000","last_fullsized":"2024-04-25T01:25:19.069039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:04:27.495446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541613+0000","last_change":"2024-04-25T01:25:16.736420+0000","last_active":"2024-04-25T01:25:18.541613+0000","last_peered":"2024-04-25T01:25:18.541613+0000","last_clean":"2024-04-25T01:25:18.541613+0000","last_became_active":"2024-04-25T01:25:16.736031+0000","last_became_peered":"2024-04-25T01:25:16.736031+0000","last_unstale":"2024-04-25T01:25:18.541613+0000","last_undegraded":"2024-04-25T01:25:18.541613+0000","last_fullsized":"2024-04-25T01:25:18.541613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:33.380905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:20.073477+0000","last_change":"2024-04-25T01:25:15.982201+0000","last_active":"2024-04-25T01:25:20.073477+0000","last_peered":"2024-04-25T01:25:20.073477+0000","last_clean":"2024-04-25T01:25:20.073477+0000","last_became_active":"2024-04-25T01:25:15.981820+0000","last_became_peered":"2024-04-25T01:25:15.981820+0000","last_unstale":"2024-04-25T01:25:20.073477+0000","last_undegraded":"2024-04-25T01:25:20.073477+0000","last_fullsized":"2024-04-25T01:25:20.073477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:12.149417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541612+0000","last_change":"2024-04-25T01:25:16.739126+0000","last_active":"2024-04-25T01:25:18.541612+0000","last_peered":"2024-04-25T01:25:18.541612+0000","last_clean":"2024-04-25T01:25:18.541612+0000","last_became_active":"2024-04-25T01:25:16.738762+0000","last_became_peered":"2024-04-25T01:25:16.738762+0000","last_unstale":"2024-04-25T01:25:18.541612+0000","last_undegraded":"2024-04-25T01:25:18.541612+0000","last_fullsized":"2024-04-25T01:25:18.541612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:00:47.661399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.440463+0000","last_change":"2024-04-25T01:25:15.985732+0000","last_active":"2024-04-25T01:25:18.440463+0000","last_peered":"2024-04-25T01:25:18.440463+0000","last_clean":"2024-04-25T01:25:18.440463+0000","last_became_active":"2024-04-25T01:25:15.985300+0000","last_became_peered":"2024-04-25T01:25:15.985300+0000","last_unstale":"2024-04-25T01:25:18.440463+0000","last_undegraded":"2024-04-25T01:25:18.440463+0000","last_fullsized":"2024-04-25T01:25:18.440463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:01.180165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541598+0000","last_change":"2024-04-25T01:25:16.746639+0000","last_active":"2024-04-25T01:25:18.541598+0000","last_peered":"2024-04-25T01:25:18.541598+0000","last_clean":"2024-04-25T01:25:18.541598+0000","last_became_active":"2024-04-25T01:25:16.746291+0000","last_became_peered":"2024-04-25T01:25:16.746291+0000","last_unstale":"2024-04-25T01:25:18.541598+0000","last_undegraded":"2024-04-25T01:25:18.541598+0000","last_fullsized":"2024-04-25T01:25:18.541598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:31.346243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731564+0000","last_change":"2024-04-25T01:25:15.989193+0000","last_active":"2024-04-25T01:25:17.731564+0000","last_peered":"2024-04-25T01:25:17.731564+0000","last_clean":"2024-04-25T01:25:17.731564+0000","last_became_active":"2024-04-25T01:25:15.988811+0000","last_became_peered":"2024-04-25T01:25:15.988811+0000","last_unstale":"2024-04-25T01:25:17.731564+0000","last_undegraded":"2024-04-25T01:25:17.731564+0000","last_fullsized":"2024-04-25T01:25:17.731564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:56.030398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:18.541730+0000","last_change":"2024-04-25T01:25:16.742209+0000","last_active":"2024-04-25T01:25:18.541730+0000","last_peered":"2024-04-25T01:25:18.541730+0000","last_clean":"2024-04-25T01:25:18.541730+0000","last_became_active":"2024-04-25T01:25:16.741810+0000","last_became_peered":"2024-04-25T01:25:16.741810+0000","last_unstale":"2024-04-25T01:25:18.541730+0000","last_undegraded":"2024-04-25T01:25:18.541730+0000","last_fullsized":"2024-04-25T01:25:18.541730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:38.008108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956617+0000","last_change":"2024-04-25T01:25:16.747264+0000","last_active":"2024-04-25T01:25:17.956617+0000","last_peered":"2024-04-25T01:25:17.956617+0000","last_clean":"2024-04-25T01:25:17.956617+0000","last_became_active":"2024-04-25T01:25:16.746514+0000","last_became_peered":"2024-04-25T01:25:16.746514+0000","last_unstale":"2024-04-25T01:25:17.956617+0000","last_undegraded":"2024-04-25T01:25:17.956617+0000","last_fullsized":"2024-04-25T01:25:17.956617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:14.672062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.732307+0000","last_change":"2024-04-25T01:25:16.753382+0000","last_active":"2024-04-25T01:25:17.732307+0000","last_peered":"2024-04-25T01:25:17.732307+0000","last_clean":"2024-04-25T01:25:17.732307+0000","last_became_active":"2024-04-25T01:25:16.752907+0000","last_became_peered":"2024-04-25T01:25:16.752907+0000","last_unstale":"2024-04-25T01:25:17.732307+0000","last_undegraded":"2024-04-25T01:25:17.732307+0000","last_fullsized":"2024-04-25T01:25:17.732307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:28.184630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731975+0000","last_change":"2024-04-25T01:25:16.752000+0000","last_active":"2024-04-25T01:25:17.731975+0000","last_peered":"2024-04-25T01:25:17.731975+0000","last_clean":"2024-04-25T01:25:17.731975+0000","last_became_active":"2024-04-25T01:25:16.749443+0000","last_became_peered":"2024-04-25T01:25:16.749443+0000","last_unstale":"2024-04-25T01:25:17.731975+0000","last_undegraded":"2024-04-25T01:25:17.731975+0000","last_fullsized":"2024-04-25T01:25:17.731975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:02.287719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956047+0000","last_change":"2024-04-25T01:25:16.742510+0000","last_active":"2024-04-25T01:25:17.956047+0000","last_peered":"2024-04-25T01:25:17.956047+0000","last_clean":"2024-04-25T01:25:17.956047+0000","last_became_active":"2024-04-25T01:25:16.742162+0000","last_became_peered":"2024-04-25T01:25:16.742162+0000","last_unstale":"2024-04-25T01:25:17.956047+0000","last_undegraded":"2024-04-25T01:25:17.956047+0000","last_fullsized":"2024-04-25T01:25:17.956047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:02.924505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731635+0000","last_change":"2024-04-25T01:25:16.756224+0000","last_active":"2024-04-25T01:25:17.731635+0000","last_peered":"2024-04-25T01:25:17.731635+0000","last_clean":"2024-04-25T01:25:17.731635+0000","last_became_active":"2024-04-25T01:25:16.755868+0000","last_became_peered":"2024-04-25T01:25:16.755868+0000","last_unstale":"2024-04-25T01:25:17.731635+0000","last_undegraded":"2024-04-25T01:25:17.731635+0000","last_fullsized":"2024-04-25T01:25:17.731635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:53.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956887+0000","last_change":"2024-04-25T01:25:16.745406+0000","last_active":"2024-04-25T01:25:17.956887+0000","last_peered":"2024-04-25T01:25:17.956887+0000","last_clean":"2024-04-25T01:25:17.956887+0000","last_became_active":"2024-04-25T01:25:16.744972+0000","last_became_peered":"2024-04-25T01:25:16.744972+0000","last_unstale":"2024-04-25T01:25:17.956887+0000","last_undegraded":"2024-04-25T01:25:17.956887+0000","last_fullsized":"2024-04-25T01:25:17.956887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:13:31.598070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.732364+0000","last_change":"2024-04-25T01:25:16.745294+0000","last_active":"2024-04-25T01:25:17.732364+0000","last_peered":"2024-04-25T01:25:17.732364+0000","last_clean":"2024-04-25T01:25:17.732364+0000","last_became_active":"2024-04-25T01:25:16.743285+0000","last_became_peered":"2024-04-25T01:25:16.743285+0000","last_unstale":"2024-04-25T01:25:17.732364+0000","last_undegraded":"2024-04-25T01:25:17.732364+0000","last_fullsized":"2024-04-25T01:25:17.732364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:36.189339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955773+0000","last_change":"2024-04-25T01:25:16.755167+0000","last_active":"2024-04-25T01:25:17.955773+0000","last_peered":"2024-04-25T01:25:17.955773+0000","last_clean":"2024-04-25T01:25:17.955773+0000","last_became_active":"2024-04-25T01:25:16.754787+0000","last_became_peered":"2024-04-25T01:25:16.754787+0000","last_unstale":"2024-04-25T01:25:17.955773+0000","last_undegraded":"2024-04-25T01:25:17.955773+0000","last_fullsized":"2024-04-25T01:25:17.955773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:52.224547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956512+0000","last_change":"2024-04-25T01:25:16.737313+0000","last_active":"2024-04-25T01:25:17.956512+0000","last_peered":"2024-04-25T01:25:17.956512+0000","last_clean":"2024-04-25T01:25:17.956512+0000","last_became_active":"2024-04-25T01:25:16.736955+0000","last_became_peered":"2024-04-25T01:25:16.736955+0000","last_unstale":"2024-04-25T01:25:17.956512+0000","last_undegraded":"2024-04-25T01:25:17.956512+0000","last_fullsized":"2024-04-25T01:25:17.956512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:17.692960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.955887+0000","last_change":"2024-04-25T01:25:16.750026+0000","last_active":"2024-04-25T01:25:17.955887+0000","last_peered":"2024-04-25T01:25:17.955887+0000","last_clean":"2024-04-25T01:25:17.955887+0000","last_became_active":"2024-04-25T01:25:16.749588+0000","last_became_peered":"2024-04-25T01:25:16.749588+0000","last_unstale":"2024-04-25T01:25:17.955887+0000","last_undegraded":"2024-04-25T01:25:17.955887+0000","last_fullsized":"2024-04-25T01:25:17.955887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:09.357719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956999+0000","last_change":"2024-04-25T01:25:16.743440+0000","last_active":"2024-04-25T01:25:17.956999+0000","last_peered":"2024-04-25T01:25:17.956999+0000","last_clean":"2024-04-25T01:25:17.956999+0000","last_became_active":"2024-04-25T01:25:16.742904+0000","last_became_peered":"2024-04-25T01:25:16.742904+0000","last_unstale":"2024-04-25T01:25:17.956999+0000","last_undegraded":"2024-04-25T01:25:17.956999+0000","last_fullsized":"2024-04-25T01:25:17.956999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:17.380260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.731365+0000","last_change":"2024-04-25T01:25:16.738305+0000","last_active":"2024-04-25T01:25:17.731365+0000","last_peered":"2024-04-25T01:25:17.731365+0000","last_clean":"2024-04-25T01:25:17.731365+0000","last_became_active":"2024-04-25T01:25:16.737907+0000","last_became_peered":"2024-04-25T01:25:16.737907+0000","last_unstale":"2024-04-25T01:25:17.731365+0000","last_undegraded":"2024-04-25T01:25:17.731365+0000","last_fullsized":"2024-04-25T01:25:17.731365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:29.501388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T01:25:17.956408+0000","last_change":"2024-04-25T01:25:16.757755+0000","last_active":"2024-04-25T01:25:17.956408+0000","last_peered":"2024-04-25T01:25:17.956408+0000","last_clean":"2024-04-25T01:25:17.956408+0000","last_became_active":"2024-04-25T01:25:16.757374+0000","last_became_peered":"2024-04-25T01:25:16.757374+0000","last_unstale":"2024-04-25T01:25:17.956408+0000","last_undegraded":"2024-04-25T01:25:17.956408+0000","last_fullsized":"2024-04-25T01:25:17.956408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:46.848407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250979,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4600,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":15,"num_read_kb":12,"num_write":113,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":88563,"data_compressed":3712,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1031457,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1031072,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":476714,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":460766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":438922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1030482,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":38193,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39543,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:25:24.200 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:25:24.200 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T01:25:24.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T01:25:24.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:24 smithi038 ceph-mon[32841]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:24.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:24 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2782064559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:24.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:24 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3163627776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:24.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:24 smithi156 ceph-mon[32631]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:24.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:24 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2782064559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:25:24.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:24 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3163627776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:25.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:25 smithi016 ceph-mon[29153]: pgmap v297: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-04-25T01:25:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:25 smithi016 ceph-mon[29153]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:25 smithi016 ceph-mon[29153]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:25.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:25 smithi038 ceph-mon[32841]: pgmap v297: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-04-25T01:25:25.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:25 smithi038 ceph-mon[32841]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:25.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:25 smithi038 ceph-mon[32841]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:25.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:25 smithi156 ceph-mon[32631]: pgmap v297: 97 pgs: 97 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-04-25T01:25:25.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:25 smithi156 ceph-mon[32631]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:25.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:25 smithi156 ceph-mon[32631]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.431+0000 7f6f3c297700 1 -- 172.21.15.16:0/1501721764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 msgr2=0x7f6f34102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.431+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1501721764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34102700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f300099c0 tx=0x7f6f30009cd0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 -- 172.21.15.16:0/1501721764 shutdown_connections 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1501721764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f34103e80 0x7f6f3410a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1501721764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34102700 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1501721764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f341034e0 0x7f6f34103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 -- 172.21.15.16:0/1501721764 >> 172.21.15.16:0/1501721764 conn(0x7f6f340fd880 msgr2=0x7f6f340ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 -- 172.21.15.16:0/1501721764 shutdown_connections 2024-04-25T01:25:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 -- 172.21.15.16:0/1501721764 wait complete. 2024-04-25T01:25:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 Processor -- start 2024-04-25T01:25:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.432+0000 7f6f3c297700 1 -- start start 2024-04-25T01:25:26.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f341034e0 0x7f6f34198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f34103e80 0x7f6f3419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32962/0 (socket says 172.21.15.16:32962) 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 -- 172.21.15.16:0/1719880395 learned_addr learned my addr 172.21.15.16:0/1719880395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3410de20 con 0x7f6f341022e0 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3410dca0 con 0x7f6f341034e0 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3410dfa0 con 0x7f6f34103e80 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f39832700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f341034e0 0x7f6f34198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a834700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f34103e80 0x7f6f3419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 -- 172.21.15.16:0/1719880395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f34103e80 msgr2=0x7f6f3419f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f34103e80 0x7f6f3419f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 -- 172.21.15.16:0/1719880395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f341034e0 msgr2=0x7f6f34198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f341034e0 0x7f6f34198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3419ffa0 con 0x7f6f341022e0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3a033700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34198340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6f300038f0 tx=0x7f6f300057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30027070 con 0x7f6f341022e0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f341a0230 con 0x7f6f341022e0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.433+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f30004200 con 0x7f6f341022e0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.434+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3000f460 con 0x7f6f341022e0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.434+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f341a0700 con 0x7f6f341022e0 2024-04-25T01:25:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.434+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f340507a0 con 0x7f6f341022e0 2024-04-25T01:25:26.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.436+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6f3002a070 con 0x7f6f341022e0 2024-04-25T01:25:26.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.436+0000 7f6f2b7fe700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f200a2d60 0x7f6f200a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:26.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.436+0000 7f6f39832700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f200a2d60 0x7f6f200a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:26.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.437+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6f300cffa0 con 0x7f6f341022e0 2024-04-25T01:25:26.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.437+0000 7f6f39832700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f200a2d60 0x7f6f200a5220 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009a10 tx=0x7f6f24009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:26.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.441+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6f3005e740 con 0x7f6f341022e0 2024-04-25T01:25:26.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.732+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f6f34062e40 con 0x7f6f341022e0 2024-04-25T01:25:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.735+0000 7f6f2b7fe700 1 -- 172.21.15.16:0/1719880395 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f6f300a1e20 con 0x7f6f341022e0 2024-04-25T01:25:26.715 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-25T01:25:26.715 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAGsSlmJY3BKxAAQ84uIzfU8kqJcfhw6SO7Og== 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f200a2d60 msgr2=0x7f6f200a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f200a2d60 0x7f6f200a5220 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009a10 tx=0x7f6f24009190 comp rx=0 tx=0).stop 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 msgr2=0x7f6f34198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34198340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6f300038f0 tx=0x7f6f300057c0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 shutdown_connections 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f200a2d60 0x7f6f200a5220 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f34103e80 0x7f6f3419f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f341022e0 0x7f6f34198340 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 --2- 172.21.15.16:0/1719880395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f341034e0 0x7f6f34198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:26.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 >> 172.21.15.16:0/1719880395 conn(0x7f6f340fd880 msgr2=0x7f6f341067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 shutdown_connections 2024-04-25T01:25:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:26.737+0000 7f6f3c297700 1 -- 172.21.15.16:0/1719880395 wait complete. 2024-04-25T01:25:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:27 smithi016 ceph-mon[29153]: pgmap v298: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 9.0 KiB/s wr, 21 op/s 2024-04-25T01:25:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:27 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1719880395' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:25:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:27 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1719880395' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:25:27.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:27 smithi038 ceph-mon[32841]: pgmap v298: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 9.0 KiB/s wr, 21 op/s 2024-04-25T01:25:27.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:27 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1719880395' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:25:27.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:27 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1719880395' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:25:27.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:27 smithi156 ceph-mon[32631]: pgmap v298: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 9.0 KiB/s wr, 21 op/s 2024-04-25T01:25:27.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:27 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1719880395' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:25:27.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:27 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1719880395' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:25:27.560 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:25:27.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T01:25:27.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:25:27.600 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:25:27.600 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:25:27.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T01:25:27.889 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:29.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:29 smithi016 ceph-mon[29153]: pgmap v299: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.1 KiB/s wr, 19 op/s 2024-04-25T01:25:29.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:29 smithi038 ceph-mon[32841]: pgmap v299: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.1 KiB/s wr, 19 op/s 2024-04-25T01:25:29.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:29 smithi156 ceph-mon[32631]: pgmap v299: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.1 KiB/s wr, 19 op/s 2024-04-25T01:25:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 -- 172.21.15.16:0/2852950142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074d80 msgr2=0x7fce1410e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 --2- 172.21.15.16:0/2852950142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074d80 0x7fce1410e720 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fce04009990 tx=0x7fce04009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 -- 172.21.15.16:0/2852950142 shutdown_connections 2024-04-25T01:25:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 --2- 172.21.15.16:0/2852950142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce1410ec60 0x7fce14111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 --2- 172.21.15.16:0/2852950142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce14074460 0x7fce14074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 --2- 172.21.15.16:0/2852950142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074d80 0x7fce1410e720 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 -- 172.21.15.16:0/2852950142 >> 172.21.15.16:0/2852950142 conn(0x7fce140793a0 msgr2=0x7fce140797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 -- 172.21.15.16:0/2852950142 shutdown_connections 2024-04-25T01:25:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.779+0000 7fce1b17b700 1 -- 172.21.15.16:0/2852950142 wait complete. 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 Processor -- start 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 -- start start 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 0x7fce141a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce14074d80 0x7fce141a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce1410ec60 0x7fce1419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce14114660 con 0x7fce14074460 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fce141144e0 con 0x7fce1410ec60 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.780+0000 7fce1b17b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fce141147e0 con 0x7fce14074d80 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 0x7fce141a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 0x7fce141a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32980/0 (socket says 172.21.15.16:32980) 2024-04-25T01:25:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 -- 172.21.15.16:0/4119294476 learned_addr learned my addr 172.21.15.16:0/4119294476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce19718700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce1410ec60 0x7fce1419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce13fff700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce14074d80 0x7fce141a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 -- 172.21.15.16:0/4119294476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce14074d80 msgr2=0x7fce141a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce14074d80 0x7fce141a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 -- 172.21.15.16:0/4119294476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce1410ec60 msgr2=0x7fce1419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce1410ec60 0x7fce1419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 -- 172.21.15.16:0/4119294476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1419b140 con 0x7fce14074460 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce18f17700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 0x7fce141a07d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00d950 tx=0x7fcdfc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc016070 con 0x7fce14074460 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdfc00fd40 con 0x7fce14074460 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1419b430 con 0x7fce14074460 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.781+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00e3b0 con 0x7fce14074460 2024-04-25T01:25:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.782+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce141a76a0 con 0x7fce14074460 2024-04-25T01:25:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.783+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce14068f20 con 0x7fce14074460 2024-04-25T01:25:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.784+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcdfc019070 con 0x7fce14074460 2024-04-25T01:25:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.785+0000 7fce11ffb700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fce000a2cb0 0x7fce000a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.785+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcdfc0cee90 con 0x7fce14074460 2024-04-25T01:25:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.785+0000 7fce13fff700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fce000a2cb0 0x7fce000a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.785+0000 7fce13fff700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fce000a2cb0 0x7fce000a5170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fce0400b5c0 tx=0x7fce04005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:29.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:29.788+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdfc09be40 con 0x7fce14074460 2024-04-25T01:25:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.052+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fce1402db30 con 0x7fce14074460 2024-04-25T01:25:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.058+0000 7fce11ffb700 1 -- 172.21.15.16:0/4119294476 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263201 (secure 0 0 0) 0x7fcdfc09b740 con 0x7fce14074460 2024-04-25T01:25:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.060+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fce000a2cb0 msgr2=0x7fce000a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.060+0000 7fce1b17b700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fce000a2cb0 0x7fce000a5170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fce0400b5c0 tx=0x7fce04005960 comp rx=0 tx=0).stop 2024-04-25T01:25:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.060+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 msgr2=0x7fce141a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.060+0000 7fce1b17b700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 0x7fce141a07d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00d950 tx=0x7fcdfc00dc60 comp rx=0 tx=0).stop 2024-04-25T01:25:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 shutdown_connections 2024-04-25T01:25:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fce000a2cb0 0x7fce000a5170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce1410ec60 0x7fce1419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce14074460 0x7fce141a07d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 --2- 172.21.15.16:0/4119294476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce14074d80 0x7fce141a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 >> 172.21.15.16:0/4119294476 conn(0x7fce140793a0 msgr2=0x7fce140ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 shutdown_connections 2024-04-25T01:25:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:30.061+0000 7fce1b17b700 1 -- 172.21.15.16:0/4119294476 wait complete. 2024-04-25T01:25:30.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:30 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4119294476' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:30 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/4119294476' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:30 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/4119294476' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:25:30.911 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1051975299},{"type":"v1","addr":"172.21.15.16:6801","nonce":1051975299}]},"active_addr":"172.21.15.16:6801/1051975299","active_change":"2024-04-25T01:17:59.526839+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2861749009}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2156060995}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3919539389}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4002106571}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1791200647}]}]} 2024-04-25T01:25:30.918 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T01:25:30.919 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T01:25:30.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:25:31.174 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:31.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:31 smithi016 ceph-mon[29153]: pgmap v300: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.3 KiB/s wr, 17 op/s 2024-04-25T01:25:31.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:31 smithi038 ceph-mon[32841]: pgmap v300: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.3 KiB/s wr, 17 op/s 2024-04-25T01:25:31.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:31 smithi156 ceph-mon[32631]: pgmap v300: 97 pgs: 97 active+clean; 618 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.3 KiB/s wr, 17 op/s 2024-04-25T01:25:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 -- 172.21.15.16:0/4020472394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c1024e0 msgr2=0x7f706c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 --2- 172.21.15.16:0/4020472394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c1024e0 0x7f706c1028c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f705c009990 tx=0x7f705c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 -- 172.21.15.16:0/4020472394 shutdown_connections 2024-04-25T01:25:32.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 --2- 172.21.15.16:0/4020472394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f706c107570 0x7f706c10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 --2- 172.21.15.16:0/4020472394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c1024e0 0x7f706c1028c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 --2- 172.21.15.16:0/4020472394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f706c102e90 0x7f706c106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 -- 172.21.15.16:0/4020472394 >> 172.21.15.16:0/4020472394 conn(0x7f706c079390 msgr2=0x7f706c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 -- 172.21.15.16:0/4020472394 shutdown_connections 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.885+0000 7f7073cbf700 1 -- 172.21.15.16:0/4020472394 wait complete. 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 Processor -- start 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 -- start start 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f706c1024e0 0x7f706c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 0x7f706c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f706c107570 0x7f706c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f706c1124e0 con 0x7f706c102e90 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f706c112360 con 0x7f706c107570 2024-04-25T01:25:32.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f7073cbf700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f706c112660 con 0x7f706c1024e0 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f707125a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 0x7f706c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.886+0000 7f707125a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 0x7f706c19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53424/0 (socket says 172.21.15.16:53424) 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 -- 172.21.15.16:0/1287983385 learned_addr learned my addr 172.21.15.16:0/1287983385 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707225c700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f706c107570 0x7f706c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f7071a5b700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f706c1024e0 0x7f706c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 -- 172.21.15.16:0/1287983385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f706c1024e0 msgr2=0x7f706c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f706c1024e0 0x7f706c19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 -- 172.21.15.16:0/1287983385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f706c107570 msgr2=0x7f706c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f706c107570 0x7f706c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 -- 172.21.15.16:0/1287983385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c198fc0 con 0x7f706c102e90 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f707125a700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 0x7f706c19eba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f706800b500 tx=0x7f706800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068015020 con 0x7f706c102e90 2024-04-25T01:25:32.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7068011ce0 con 0x7f706c102e90 2024-04-25T01:25:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c1992b0 con 0x7f706c102e90 2024-04-25T01:25:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068010d40 con 0x7f706c102e90 2024-04-25T01:25:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.887+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c1a54c0 con 0x7f706c102e90 2024-04-25T01:25:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.888+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706c103970 con 0x7f706c102e90 2024-04-25T01:25:32.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.890+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7068018020 con 0x7f706c102e90 2024-04-25T01:25:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.891+0000 7f7062ffd700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70580a2cb0 0x7f70580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.891+0000 7f7071a5b700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70580a2cb0 0x7f70580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.891+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f70680cec90 con 0x7f706c102e90 2024-04-25T01:25:32.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.891+0000 7f7071a5b700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70580a2cb0 0x7f70580a5170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f705c005cd0 tx=0x7f705c005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:32.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:32.894+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f706801f020 con 0x7f706c102e90 2024-04-25T01:25:33.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.101+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f706c10a790 con 0x7f706c102e90 2024-04-25T01:25:33.082 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.102+0000 7f7062ffd700 1 -- 172.21.15.16:0/1287983385 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22196 (secure 0 0 0) 0x7f706809bc50 con 0x7f706c102e90 2024-04-25T01:25:33.082 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:33.083 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:25:17.723944+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70580a2cb0 msgr2=0x7f70580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70580a2cb0 0x7f70580a5170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f705c005cd0 tx=0x7f705c005c40 comp rx=0 tx=0).stop 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 msgr2=0x7f706c19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 0x7f706c19eba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f706800b500 tx=0x7f706800b810 comp rx=0 tx=0).stop 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 shutdown_connections 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70580a2cb0 0x7f70580a5170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f706c107570 0x7f706c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.104+0000 7f7073cbf700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f706c1024e0 0x7f706c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.105+0000 7f7073cbf700 1 --2- 172.21.15.16:0/1287983385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706c102e90 0x7f706c19eba0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.105+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 >> 172.21.15.16:0/1287983385 conn(0x7f706c079390 msgr2=0x7f706c100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.105+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 shutdown_connections 2024-04-25T01:25:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:33.105+0000 7f7073cbf700 1 -- 172.21.15.16:0/1287983385 wait complete. 2024-04-25T01:25:33.119 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:33 smithi016 ceph-mon[29153]: pgmap v301: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.5 KiB/s wr, 18 op/s 2024-04-25T01:25:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:33 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1287983385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:33.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:33 smithi038 ceph-mon[32841]: pgmap v301: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.5 KiB/s wr, 18 op/s 2024-04-25T01:25:33.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:33 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1287983385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:33.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:33 smithi156 ceph-mon[32631]: pgmap v301: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.5 KiB/s wr, 18 op/s 2024-04-25T01:25:33.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:33 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1287983385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:33.798 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T01:25:33.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:25:34.053 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:35.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:35 smithi016 ceph-mon[29153]: pgmap v302: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.6 KiB/s wr, 15 op/s 2024-04-25T01:25:35.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:35 smithi038 ceph-mon[32841]: pgmap v302: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.6 KiB/s wr, 15 op/s 2024-04-25T01:25:35.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:35 smithi156 ceph-mon[32631]: pgmap v302: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.6 KiB/s wr, 15 op/s 2024-04-25T01:25:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 -- 172.21.15.16:0/2321114422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed8107610 msgr2=0x7f9ed810f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 --2- 172.21.15.16:0/2321114422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed8107610 0x7f9ed810f190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000b0d0 tx=0x7f9ed000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:25:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 -- 172.21.15.16:0/2321114422 shutdown_connections 2024-04-25T01:25:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 --2- 172.21.15.16:0/2321114422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed8107610 0x7f9ed810f190 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 --2- 172.21.15.16:0/2321114422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed806ac00 0x7f9ed806afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 --2- 172.21.15.16:0/2321114422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806b5b0 0x7f9ed8106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 -- 172.21.15.16:0/2321114422 >> 172.21.15.16:0/2321114422 conn(0x7f9ed8077180 msgr2=0x7f9ed8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 -- 172.21.15.16:0/2321114422 shutdown_connections 2024-04-25T01:25:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.854+0000 7f9edf751700 1 -- 172.21.15.16:0/2321114422 wait complete. 2024-04-25T01:25:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 Processor -- start 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 -- start start 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 0x7f9ed8101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed806b5b0 0x7f9ed81017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8107610 0x7f9ed8101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8112480 con 0x7f9ed806b5b0 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edd4ed700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 0x7f9ed8101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edd4ed700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 0x7f9ed8101290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45790/0 (socket says 172.21.15.16:45790) 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edd4ed700 1 -- 172.21.15.16:0/853863274 learned_addr learned my addr 172.21.15.16:0/853863274 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8112300 con 0x7f9ed8107610 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8112600 con 0x7f9ed806ac00 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9eddcee700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8107610 0x7f9ed8101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.855+0000 7f9edccec700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed806b5b0 0x7f9ed81017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edd4ed700 1 -- 172.21.15.16:0/853863274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8107610 msgr2=0x7f9ed8101d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edd4ed700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8107610 0x7f9ed8101d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edd4ed700 1 -- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed806b5b0 msgr2=0x7f9ed81017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edd4ed700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed806b5b0 0x7f9ed81017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edd4ed700 1 -- 172.21.15.16:0/853863274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed81a6e60 con 0x7f9ed806ac00 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edd4ed700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 0x7f9ed8101290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400d950 tx=0x7f9ed400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4016070 con 0x7f9ed806ac00 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ed400fd40 con 0x7f9ed806ac00 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed81a7150 con 0x7f9ed806ac00 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400e3b0 con 0x7f9ed806ac00 2024-04-25T01:25:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.856+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed81a76e0 con 0x7f9ed806ac00 2024-04-25T01:25:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.857+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed8068f20 con 0x7f9ed806ac00 2024-04-25T01:25:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.859+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9ed4019070 con 0x7f9ed806ac00 2024-04-25T01:25:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.859+0000 7f9ece7fc700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ec40a2d30 0x7f9ec40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.860+0000 7f9edccec700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ec40a2d30 0x7f9ec40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.860+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9ed40ceb10 con 0x7f9ed806ac00 2024-04-25T01:25:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.860+0000 7f9edccec700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ec40a2d30 0x7f9ec40a51f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9ed81070d0 tx=0x7f9ec800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:35.863+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9ed409b9c0 con 0x7f9ed806ac00 2024-04-25T01:25:36.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.069+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ed802db30 con 0x7f9ed806ac00 2024-04-25T01:25:36.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.071+0000 7f9ece7fc700 1 -- 172.21.15.16:0/853863274 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22196 (secure 0 0 0) 0x7f9ed409b2c0 con 0x7f9ed806ac00 2024-04-25T01:25:36.051 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:25:36.051 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:25:17.723944+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ec40a2d30 msgr2=0x7f9ec40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ec40a2d30 0x7f9ec40a51f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9ed81070d0 tx=0x7f9ec800a300 comp rx=0 tx=0).stop 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 msgr2=0x7f9ed8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 0x7f9ed8101290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400d950 tx=0x7f9ed400dc60 comp rx=0 tx=0).stop 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 shutdown_connections 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9ec40a2d30 0x7f9ec40a51f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8107610 0x7f9ed8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ed806ac00 0x7f9ed8101290 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 --2- 172.21.15.16:0/853863274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed806b5b0 0x7f9ed81017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 >> 172.21.15.16:0/853863274 conn(0x7f9ed8077180 msgr2=0x7f9ed8108740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 shutdown_connections 2024-04-25T01:25:36.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:36.073+0000 7f9edf751700 1 -- 172.21.15.16:0/853863274 wait complete. 2024-04-25T01:25:36.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:36 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/853863274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:36.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:36 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/853863274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:36.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:36 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/853863274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:25:36.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T01:25:36.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T01:25:36.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T01:25:36.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T01:25:36.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T01:25:36.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T01:25:36.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T01:25:36.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T01:25:36.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T01:25:36.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T01:25:36.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T01:25:36.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T01:25:37.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:37 smithi038 ceph-mon[32841]: pgmap v303: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-04-25T01:25:37.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:37 smithi156 ceph-mon[32631]: pgmap v303: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-04-25T01:25:37.570 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:37.572 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:37.578 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:37 smithi016 ceph-mon[29153]: pgmap v303: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-04-25T01:25:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.506 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.507 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.508 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.509 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.509 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:38.511 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:39.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:39 smithi016 ceph-mon[29153]: pgmap v304: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-25T01:25:39.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:39 smithi038 ceph-mon[32841]: pgmap v304: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-25T01:25:39.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:39 smithi156 ceph-mon[32631]: pgmap v304: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-25T01:25:41.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:41 smithi016 ceph-mon[29153]: pgmap v305: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-25T01:25:41.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:41 smithi038 ceph-mon[32841]: pgmap v305: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-25T01:25:41.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:41 smithi156 ceph-mon[32631]: pgmap v305: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-25T01:25:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:43.444 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:43 smithi016 ceph-mon[29153]: pgmap v306: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:25:43.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:43 smithi038 ceph-mon[32841]: pgmap v306: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:25:43.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:43 smithi156 ceph-mon[32631]: pgmap v306: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:25:43.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1211153339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 msgr2=0x7f4b64106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1211153339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b64106e90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009990 tx=0x7f4b58009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1211153339 shutdown_connections 2024-04-25T01:25:43.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1211153339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b64107570 0x7f4b6410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1211153339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b641024e0 0x7f4b641028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1211153339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b64106e90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1211153339 >> 172.21.15.16:0/1211153339 conn(0x7f4b64079390 msgr2=0x7f4b640797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:43.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1211153339 shutdown_connections 2024-04-25T01:25:43.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.909+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1211153339 wait complete. 2024-04-25T01:25:43.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 Processor -- start 2024-04-25T01:25:43.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 -- start start 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b641024e0 0x7f4b641a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b641a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b64107570 0x7f4b6419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b641123e0 con 0x7f4b64102e90 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b64112260 con 0x7f4b641024e0 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b6b3ea700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4b64112560 con 0x7f4b64107570 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b68985700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b641a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b68985700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b641a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53346/0 (socket says 172.21.15.16:53346) 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b68985700 1 -- 172.21.15.16:0/1306970300 learned_addr learned my addr 172.21.15.16:0/1306970300 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b69186700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b641024e0 0x7f4b641a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.910+0000 7f4b69987700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b64107570 0x7f4b6419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b68985700 1 -- 172.21.15.16:0/1306970300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b64107570 msgr2=0x7f4b6419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b68985700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b64107570 0x7f4b6419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b68985700 1 -- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b641024e0 msgr2=0x7f4b641a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b68985700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b641024e0 0x7f4b641a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b68985700 1 -- 172.21.15.16:0/1306970300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6419b180 con 0x7f4b64102e90 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b69186700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b641024e0 0x7f4b641a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b69987700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b64107570 0x7f4b6419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b68985700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b641a0d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4b58000c00 tx=0x7f4b58003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58027070 con 0x7f4b64102e90 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b5800bdc0 con 0x7f4b64102e90 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6419b410 con 0x7f4b64102e90 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58022d60 con 0x7f4b64102e90 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.911+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b641a76e0 con 0x7f4b64102e90 2024-04-25T01:25:43.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.912+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b6410e7f0 con 0x7f4b64102e90 2024-04-25T01:25:43.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.913+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4b5802a070 con 0x7f4b64102e90 2024-04-25T01:25:43.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b627fc700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b4c0a2cb0 0x7f4b4c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b69186700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b4c0a2cb0 0x7f4b4c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4b580d0460 con 0x7f4b64102e90 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b627fc700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] conn(0x7f4b4c0ac490 0x7f4b4c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 --> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b58001d30 con 0x7f4b4c0ac490 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4b580d0810 con 0x7f4b64102e90 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.914+0000 7f4b69987700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] conn(0x7f4b4c0ac490 0x7f4b4c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.915+0000 7f4b69186700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b4c0a2cb0 0x7f4b4c0a5170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c007fd0 tx=0x7f4b5c00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.915+0000 7f4b69987700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] conn(0x7f4b4c0ac490 0x7f4b4c0ae8b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:43.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.916+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== osd.1 v2:172.21.15.38:6800/4146278985 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b58001d30 con 0x7f4b4c0ac490 2024-04-25T01:25:43.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.932+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 --> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b64103970 con 0x7f4b4c0ac490 2024-04-25T01:25:43.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.933+0000 7f4b627fc700 1 -- 172.21.15.16:0/1306970300 <== osd.1 v2:172.21.15.38:6800/4146278985 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4b64103970 con 0x7f4b4c0ac490 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.933+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] conn(0x7f4b4c0ac490 msgr2=0x7f4b4c0ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.933+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] conn(0x7f4b4c0ac490 0x7f4b4c0ae8b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.933+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b4c0a2cb0 msgr2=0x7f4b4c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.933+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4b4c0a2cb0 0x7f4b4c0a5170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c007fd0 tx=0x7f4b5c00c040 comp rx=0 tx=0).stop 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 msgr2=0x7f4b641a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b6b3ea700 1 --2- 172.21.15.16:0/1306970300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b64102e90 0x7f4b641a0d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4b58000c00 tx=0x7f4b58003a80 comp rx=0 tx=0).stop 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b69987700 1 -- 172.21.15.16:0/1306970300 reap_dead start 2024-04-25T01:25:43.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 shutdown_connections 2024-04-25T01:25:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 >> 172.21.15.16:0/1306970300 conn(0x7f4b64079390 msgr2=0x7f4b641086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 shutdown_connections 2024-04-25T01:25:43.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:43.934+0000 7f4b6b3ea700 1 -- 172.21.15.16:0/1306970300 wait complete. 2024-04-25T01:25:45.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:45 smithi038 ceph-mon[32841]: pgmap v307: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T01:25:45.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:45 smithi156 ceph-mon[32631]: pgmap v307: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T01:25:45.519 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:45 smithi016 ceph-mon[29153]: pgmap v307: 97 pgs: 97 active+clean; 635 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T01:25:47.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:47 smithi016 ceph-mon[29153]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:25:47.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:47 smithi038 ceph-mon[32841]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:25:47.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:47 smithi156 ceph-mon[32631]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T01:25:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 -- 172.21.15.16:0/2670631075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 msgr2=0x7f96a410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 --2- 172.21.15.16:0/2670631075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a410f0f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f969c00b0a0 tx=0x7f969c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 -- 172.21.15.16:0/2670631075 shutdown_connections 2024-04-25T01:25:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 --2- 172.21.15.16:0/2670631075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a410f0f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 --2- 172.21.15.16:0/2670631075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96a41082b0 0x7f96a4108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 --2- 172.21.15.16:0/2670631075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96a4069f40 0x7f96a406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 -- 172.21.15.16:0/2670631075 >> 172.21.15.16:0/2670631075 conn(0x7f96a40780e0 msgr2=0x7f96a40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 -- 172.21.15.16:0/2670631075 shutdown_connections 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.219+0000 7f96a8927700 1 -- 172.21.15.16:0/2670631075 wait complete. 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 Processor -- start 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 -- start start 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96a4069f40 0x7f96a419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96a41082b0 0x7f96a4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f96a41124b0 con 0x7f96a406a8e0 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96a4112330 con 0x7f96a41082b0 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a8927700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f96a4112630 con 0x7f96a4069f40 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a419eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53358/0 (socket says 172.21.15.16:53358) 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 -- 172.21.15.16:0/2959731335 learned_addr learned my addr 172.21.15.16:0/2959731335 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a259c700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96a4069f40 0x7f96a419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a2d9d700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96a41082b0 0x7f96a4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 -- 172.21.15.16:0/2959731335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96a4069f40 msgr2=0x7f96a419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96a4069f40 0x7f96a419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 -- 172.21.15.16:0/2959731335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96a41082b0 msgr2=0x7f96a4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96a41082b0 0x7f96a4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 -- 172.21.15.16:0/2959731335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a4198ff0 con 0x7f96a406a8e0 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a259c700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96a4069f40 0x7f96a419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.220+0000 7f96a1d9b700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a419eb80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f969800b530 tx=0x7f969800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.221+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698015020 con 0x7f96a406a8e0 2024-04-25T01:25:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.221+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a41992e0 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.221+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9698011ce0 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.221+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698010e40 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.221+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a41a55c0 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.221+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f96a4062980 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.223+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9698018020 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.223+0000 7f96937fe700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f968c0a2cb0 0x7f968c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96a259c700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f968c0a2cb0 0x7f968c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f96980ceef0 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96937fe700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] conn(0x7f968c0ac490 0x7f968c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 --> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f968c0aef60 con 0x7f968c0ac490 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f96980cf3d0 con 0x7f96a406a8e0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96a2d9d700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] conn(0x7f968c0ac490 0x7f968c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96a259c700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f968c0a2cb0 0x7f968c0a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9694005fd0 tx=0x7f9694005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.224+0000 7f96a2d9d700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] conn(0x7f968c0ac490 0x7f968c0ae8b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.225+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== osd.6 v2:172.21.15.16:6818/2018477454 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f968c0aef60 con 0x7f968c0ac490 2024-04-25T01:25:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 --> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f96a4062e00 con 0x7f968c0ac490 2024-04-25T01:25:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96937fe700 1 -- 172.21.15.16:0/2959731335 <== osd.6 v2:172.21.15.16:6818/2018477454 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f96a4062e00 con 0x7f968c0ac490 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] conn(0x7f968c0ac490 msgr2=0x7f968c0ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] conn(0x7f968c0ac490 0x7f968c0ae8b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f968c0a2cb0 msgr2=0x7f968c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f968c0a2cb0 0x7f968c0a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9694005fd0 tx=0x7f9694005e30 comp rx=0 tx=0).stop 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 msgr2=0x7f96a419eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a8927700 1 --2- 172.21.15.16:0/2959731335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96a406a8e0 0x7f96a419eb80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f969800b530 tx=0x7f969800b840 comp rx=0 tx=0).stop 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.242+0000 7f96a2d9d700 1 -- 172.21.15.16:0/2959731335 reap_dead start 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.243+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 shutdown_connections 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.243+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 >> 172.21.15.16:0/2959731335 conn(0x7f96a40780e0 msgr2=0x7f96a40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.243+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 shutdown_connections 2024-04-25T01:25:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:49.243+0000 7f96a8927700 1 -- 172.21.15.16:0/2959731335 wait complete. 2024-04-25T01:25:49.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:49 smithi016 ceph-mon[29153]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:49.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:49 smithi038 ceph-mon[32841]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:49.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:49 smithi156 ceph-mon[32631]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:51.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:51 smithi016 ceph-mon[29153]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:51.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:51 smithi038 ceph-mon[32841]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:51.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:51 smithi156 ceph-mon[32631]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:51.857 INFO:teuthology.orchestra.run.smithi016.stdout:55834574920 2024-04-25T01:25:51.858 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T01:25:53.304 INFO:teuthology.orchestra.run.smithi016.stdout:197568495663 2024-04-25T01:25:53.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T01:25:53.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:53 smithi016 ceph-mon[29153]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:25:53.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:53 smithi038 ceph-mon[32841]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:25:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 -- 172.21.15.16:0/1571065023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 msgr2=0x7f9a2410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 --2- 172.21.15.16:0/1571065023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a2410f0f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00b0a0 tx=0x7f9a1c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 -- 172.21.15.16:0/1571065023 shutdown_connections 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 --2- 172.21.15.16:0/1571065023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a2410f0f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 --2- 172.21.15.16:0/1571065023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a241082b0 0x7f9a24108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 --2- 172.21.15.16:0/1571065023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a24069f40 0x7f9a2406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 -- 172.21.15.16:0/1571065023 >> 172.21.15.16:0/1571065023 conn(0x7f9a240780e0 msgr2=0x7f9a240784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 -- 172.21.15.16:0/1571065023 shutdown_connections 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.519+0000 7f9a29252700 1 -- 172.21.15.16:0/1571065023 wait complete. 2024-04-25T01:25:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 Processor -- start 2024-04-25T01:25:53.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:53 smithi156 ceph-mon[32631]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:25:53.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 -- start start 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a24069f40 0x7f9a24101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a24101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a241082b0 0x7f9a24101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241123e0 con 0x7f9a2406a8e0 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24112260 con 0x7f9a241082b0 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.520+0000 7f9a29252700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24112560 con 0x7f9a24069f40 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a24101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a24101740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58954/0 (socket says 172.21.15.16:58954) 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 -- 172.21.15.16:0/2919783963 learned_addr learned my addr 172.21.15.16:0/2919783963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a22d9d700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a24069f40 0x7f9a24101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2359e700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a241082b0 0x7f9a24101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 -- 172.21.15.16:0/2919783963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a24069f40 msgr2=0x7f9a24101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:53.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a24069f40 0x7f9a24101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 -- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a241082b0 msgr2=0x7f9a24101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a241082b0 0x7f9a24101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 -- 172.21.15.16:0/2919783963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a241a6e20 con 0x7f9a2406a8e0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a2259c700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a24101740 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800d980 tx=0x7f9a1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a18017070 con 0x7f9a2406a8e0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.521+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a18005020 con 0x7f9a2406a8e0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.522+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a241a7110 con 0x7f9a2406a8e0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.522+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1800fa90 con 0x7f9a2406a8e0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.522+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a241a7600 con 0x7f9a2406a8e0 2024-04-25T01:25:53.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.522+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9a24062940 con 0x7f9a2406a8e0 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.524+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9a180051d0 con 0x7f9a2406a8e0 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.524+0000 7f9a0bfff700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9a0c0a2cb0 0x7f9a0c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9a180cf470 con 0x7f9a2406a8e0 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a22d9d700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9a0c0a2cb0 0x7f9a0c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a0bfff700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] conn(0x7f9a0c0ac490 0x7f9a0c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 --> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a1801de70 con 0x7f9a0c0ac490 2024-04-25T01:25:53.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f9a180cf950 con 0x7f9a2406a8e0 2024-04-25T01:25:53.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a22d9d700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9a0c0a2cb0 0x7f9a0c0a5170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a14005950 tx=0x7f9a1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:53.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a2359e700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] conn(0x7f9a0c0ac490 0x7f9a0c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:53.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.525+0000 7f9a2359e700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] conn(0x7f9a0c0ac490 0x7f9a0c0ae8b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:53.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.526+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== osd.7 v2:172.21.15.38:6816/2653364257 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a1801de70 con 0x7f9a0c0ac490 2024-04-25T01:25:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.545+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 --> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9a24062d40 con 0x7f9a0c0ac490 2024-04-25T01:25:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a0bfff700 1 -- 172.21.15.16:0/2919783963 <== osd.7 v2:172.21.15.38:6816/2653364257 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9a24062d40 con 0x7f9a0c0ac490 2024-04-25T01:25:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] conn(0x7f9a0c0ac490 msgr2=0x7f9a0c0ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] conn(0x7f9a0c0ac490 0x7f9a0c0ae8b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9a0c0a2cb0 msgr2=0x7f9a0c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9a0c0a2cb0 0x7f9a0c0a5170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a14005950 tx=0x7f9a1400b410 comp rx=0 tx=0).stop 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 msgr2=0x7f9a24101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 --2- 172.21.15.16:0/2919783963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a2406a8e0 0x7f9a24101740 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800d980 tx=0x7f9a1800dc90 comp rx=0 tx=0).stop 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a2359e700 1 -- 172.21.15.16:0/2919783963 reap_dead start 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 shutdown_connections 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 >> 172.21.15.16:0/2919783963 conn(0x7f9a240780e0 msgr2=0x7f9a240ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 shutdown_connections 2024-04-25T01:25:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:53.546+0000 7f9a29252700 1 -- 172.21.15.16:0/2919783963 wait complete. 2024-04-25T01:25:55.293 INFO:teuthology.orchestra.run.smithi016.stdout:219043332138 2024-04-25T01:25:55.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T01:25:55.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:55 smithi038 ceph-mon[32841]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 -- 172.21.15.16:0/1154181233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc1064c0 msgr2=0x7f3efc10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 --2- 172.21.15.16:0/1154181233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc1064c0 0x7f3efc10e060 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400b0d0 tx=0x7f3ef400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 -- 172.21.15.16:0/1154181233 shutdown_connections 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 --2- 172.21.15.16:0/1154181233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc1064c0 0x7f3efc10e060 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 --2- 172.21.15.16:0/1154181233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3efc101770 0x7f3efc101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 --2- 172.21.15.16:0/1154181233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3efc102120 0x7f3efc105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 -- 172.21.15.16:0/1154181233 >> 172.21.15.16:0/1154181233 conn(0x7f3efc0780e0 msgr2=0x7f3efc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 -- 172.21.15.16:0/1154181233 shutdown_connections 2024-04-25T01:25:55.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 -- 172.21.15.16:0/1154181233 wait complete. 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.842+0000 7f3f018a8700 1 Processor -- start 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 -- start start 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc101770 0x7f3efc19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3efc102120 0x7f3efc19db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3efc1064c0 0x7f3efc197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc111400 con 0x7f3efc101770 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc111280 con 0x7f3efc102120 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3f018a8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc111580 con 0x7f3efc1064c0 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc101770 0x7f3efc19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc101770 0x7f3efc19d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58938/0 (socket says 172.21.15.16:58938) 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 -- 172.21.15.16:0/2454978542 learned_addr learned my addr 172.21.15.16:0/2454978542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efa7fc700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3efc102120 0x7f3efc19db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:55.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efb7fe700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3efc1064c0 0x7f3efc197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 -- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3efc1064c0 msgr2=0x7f3efc197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3efc1064c0 0x7f3efc197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 -- 172.21.15.16:0/2454978542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3efc102120 msgr2=0x7f3efc19db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3efc102120 0x7f3efc19db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efaffd700 1 -- 172.21.15.16:0/2454978542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc197f60 con 0x7f3efc101770 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.843+0000 7f3efb7fe700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3efc1064c0 0x7f3efc197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3efaffd700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc101770 0x7f3efc19d600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00d9e0 tx=0x7f3eec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec01b070 con 0x7f3efc101770 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3eec00ec50 con 0x7f3efc101770 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec0145a0 con 0x7f3efc101770 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc198250 con 0x7f3efc101770 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc06a900 con 0x7f3efc101770 2024-04-25T01:25:55.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.844+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3efc06af10 con 0x7f3efc101770 2024-04-25T01:25:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.846+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3eec01e070 con 0x7f3efc101770 2024-04-25T01:25:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.846+0000 7f3f008a6700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3ee40a2cb0 0x7f3ee40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.846+0000 7f3efa7fc700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3ee40a2cb0 0x7f3ee40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.846+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3eec0d0d70 con 0x7f3efc101770 2024-04-25T01:25:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.847+0000 7f3f008a6700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] conn(0x7f3ee40ac490 0x7f3ee40ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:55.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.847+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 --> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3eec011ec0 con 0x7f3ee40ac490 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.847+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f3eec0d1120 con 0x7f3efc101770 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.847+0000 7f3efa7fc700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3ee40a2cb0 0x7f3ee40a5170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3efc101150 tx=0x7f3ef0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.847+0000 7f3efb7fe700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] conn(0x7f3ee40ac490 0x7f3ee40ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.847+0000 7f3efb7fe700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] conn(0x7f3ee40ac490 0x7f3ee40ae8b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.848+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== osd.2 v2:172.21.15.156:6800/380124115 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3eec011ec0 con 0x7f3ee40ac490 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.864+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 --> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3efc06b390 con 0x7f3ee40ac490 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f008a6700 1 -- 172.21.15.16:0/2454978542 <== osd.2 v2:172.21.15.156:6800/380124115 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3efc06b390 con 0x7f3ee40ac490 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] conn(0x7f3ee40ac490 msgr2=0x7f3ee40ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] conn(0x7f3ee40ac490 0x7f3ee40ae8b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3ee40a2cb0 msgr2=0x7f3ee40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3ee40a2cb0 0x7f3ee40a5170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3efc101150 tx=0x7f3ef0009190 comp rx=0 tx=0).stop 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc101770 msgr2=0x7f3efc19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 --2- 172.21.15.16:0/2454978542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3efc101770 0x7f3efc19d600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00d9e0 tx=0x7f3eec00dcf0 comp rx=0 tx=0).stop 2024-04-25T01:25:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3efb7fe700 1 -- 172.21.15.16:0/2454978542 reap_dead start 2024-04-25T01:25:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 shutdown_connections 2024-04-25T01:25:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 >> 172.21.15.16:0/2454978542 conn(0x7f3efc0780e0 msgr2=0x7f3efc0ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 shutdown_connections 2024-04-25T01:25:55.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:52.865+0000 7f3f018a8700 1 -- 172.21.15.16:0/2454978542 wait complete. 2024-04-25T01:25:55.484 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:55 smithi016 ceph-mon[29153]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:55.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:55 smithi156 ceph-mon[32631]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:25:56.148 INFO:teuthology.orchestra.run.smithi016.stdout:77309411397 2024-04-25T01:25:56.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3561006762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04074d80 msgr2=0x7f2d04112ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3561006762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04074d80 0x7f2d04112ec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009990 tx=0x7f2cf4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3561006762 shutdown_connections 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3561006762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d04113400 0x7f2d041157f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3561006762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d04074460 0x7f2d04074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3561006762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04074d80 0x7f2d04112ec0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3561006762 >> 172.21.15.16:0/3561006762 conn(0x7f2d04079390 msgr2=0x7f2d040797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3561006762 shutdown_connections 2024-04-25T01:25:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.991+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3561006762 wait complete. 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 Processor -- start 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 -- start start 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d04074460 0x7f2d0419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d04074d80 0x7f2d0419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04113400 0x7f2d041a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04118a40 con 0x7f2d04113400 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d041188c0 con 0x7f2d04074460 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.992+0000 7f2d0ba57700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04118bc0 con 0x7f2d04074d80 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04113400 0x7f2d041a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04113400 0x7f2d041a37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58970/0 (socket says 172.21.15.16:58970) 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 -- 172.21.15.16:0/3687832259 learned_addr learned my addr 172.21.15.16:0/3687832259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d08ff2700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d04074d80 0x7f2d0419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d097f3700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d04074460 0x7f2d0419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:56.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 -- 172.21.15.16:0/3687832259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d04074d80 msgr2=0x7f2d0419f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d04074d80 0x7f2d0419f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 -- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d04074460 msgr2=0x7f2d0419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d04074460 0x7f2d0419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 -- 172.21.15.16:0/3687832259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d041a3f20 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d09ff4700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04113400 0x7f2d041a37e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc007d40 tx=0x7f2cfc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc016020 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cfc00ad40 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d041a4210 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0115e0 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.993+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d041aba90 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.994+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d0410f900 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.995+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2cfc019020 con 0x7f2d04113400 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2cfa7fc700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2cf00a2d30 0x7f2cf00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:56.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2d097f3700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2cf00a2d30 0x7f2cf00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2cfc0cf1b0 con 0x7f2d04113400 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2cfa7fc700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] conn(0x7f2cf00ac510 0x7f2cf00ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 --> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cfc01ce50 con 0x7f2cf00ac510 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f2cfc0cf690 con 0x7f2d04113400 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.996+0000 7f2d097f3700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2cf00a2d30 0x7f2cf00a51f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2d00005950 tx=0x7f2d000058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.997+0000 7f2d08ff2700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] conn(0x7f2cf00ac510 0x7f2cf00ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.997+0000 7f2d08ff2700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] conn(0x7f2cf00ac510 0x7f2cf00ae930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:56.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:56.998+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== osd.4 v2:172.21.15.38:6808/1109232541 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2cfc01ce50 con 0x7f2cf00ac510 2024-04-25T01:25:56.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.014+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 --> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d04075b50 con 0x7f2cf00ac510 2024-04-25T01:25:56.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2cfa7fc700 1 -- 172.21.15.16:0/3687832259 <== osd.4 v2:172.21.15.38:6808/1109232541 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2d04075b50 con 0x7f2cf00ac510 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] conn(0x7f2cf00ac510 msgr2=0x7f2cf00ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] conn(0x7f2cf00ac510 0x7f2cf00ae930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2cf00a2d30 msgr2=0x7f2cf00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2cf00a2d30 0x7f2cf00a51f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2d00005950 tx=0x7f2d000058e0 comp rx=0 tx=0).stop 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04113400 msgr2=0x7f2d041a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 --2- 172.21.15.16:0/3687832259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d04113400 0x7f2d041a37e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc007d40 tx=0x7f2cfc00a340 comp rx=0 tx=0).stop 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d09ff4700 1 -- 172.21.15.16:0/3687832259 reap_dead start 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 shutdown_connections 2024-04-25T01:25:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 >> 172.21.15.16:0/3687832259 conn(0x7f2d04079390 msgr2=0x7f2d0410eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 shutdown_connections 2024-04-25T01:25:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.015+0000 7f2d0ba57700 1 -- 172.21.15.16:0/3687832259 wait complete. 2024-04-25T01:25:57.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 -- 172.21.15.16:0/3218948695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd806a8e0 msgr2=0x7f4cd810f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3218948695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd806a8e0 0x7f4cd810f0f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b0a0 tx=0x7f4cd000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 -- 172.21.15.16:0/3218948695 shutdown_connections 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3218948695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd806a8e0 0x7f4cd810f0f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3218948695 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd81082b0 0x7f4cd8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3218948695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd8069f40 0x7f4cd806a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 -- 172.21.15.16:0/3218948695 >> 172.21.15.16:0/3218948695 conn(0x7f4cd80780e0 msgr2=0x7f4cd80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 -- 172.21.15.16:0/3218948695 shutdown_connections 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.220+0000 7f4cde81c700 1 -- 172.21.15.16:0/3218948695 wait complete. 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 Processor -- start 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 -- start start 2024-04-25T01:25:57.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8069f40 0x7f4cd81a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd806a8e0 0x7f4cd81a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81082b0 0x7f4cd819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8112460 con 0x7f4cd81082b0 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81122e0 con 0x7f4cd806a8e0 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cde81c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81125e0 con 0x7f4cd8069f40 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81082b0 0x7f4cd819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81082b0 0x7f4cd819a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58992/0 (socket says 172.21.15.16:58992) 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 -- 172.21.15.16:0/3704563034 learned_addr learned my addr 172.21.15.16:0/3704563034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cd7fff700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8069f40 0x7f4cd81a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 -- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8069f40 msgr2=0x7f4cd81a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cd8069f40 0x7f4cd81a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 -- 172.21.15.16:0/3704563034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd806a8e0 msgr2=0x7f4cd81a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cd77fe700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd806a8e0 0x7f4cd81a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.221+0000 7f4cdcdb9700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd806a8e0 0x7f4cd81a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cdcdb9700 1 -- 172.21.15.16:0/3704563034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd819b0f0 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cd77fe700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cd806a8e0 0x7f4cd81a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cdcdb9700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81082b0 0x7f4cd819a990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b830 tx=0x7f4cd0004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0021070 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd0007e30 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd819b380 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd00047d0 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.222+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd81a76a0 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.223+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4cd810df10 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.224+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4cd0007980 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd57fa700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc00a2d30 0x7f4cc00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd7fff700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc00a2d30 0x7f4cc00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4cd000f020 con 0x7f4cd81082b0 2024-04-25T01:25:57.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd57fa700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] conn(0x7f4cc00ac510 0x7f4cc00ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 --> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cd001ce50 con 0x7f4cc00ac510 2024-04-25T01:25:57.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4cd00d0bf0 con 0x7f4cd81082b0 2024-04-25T01:25:57.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd7fff700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc00a2d30 0x7f4cc00a51f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4cd819bad0 tx=0x7f4cc800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:57.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.225+0000 7f4cd77fe700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] conn(0x7f4cc00ac510 0x7f4cc00ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.226+0000 7f4cd77fe700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] conn(0x7f4cc00ac510 0x7f4cc00ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:57.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.226+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== osd.5 v2:172.21.15.156:6808/3440527374 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4cd001ce50 con 0x7f4cc00ac510 2024-04-25T01:25:57.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.243+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 --> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4cd8103960 con 0x7f4cc00ac510 2024-04-25T01:25:57.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.243+0000 7f4cd57fa700 1 -- 172.21.15.16:0/3704563034 <== osd.5 v2:172.21.15.156:6808/3440527374 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4cd8103960 con 0x7f4cc00ac510 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] conn(0x7f4cc00ac510 msgr2=0x7f4cc00ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] conn(0x7f4cc00ac510 0x7f4cc00ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc00a2d30 msgr2=0x7f4cc00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4cc00a2d30 0x7f4cc00a51f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4cd819bad0 tx=0x7f4cc800b3c0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81082b0 msgr2=0x7f4cd819a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 --2- 172.21.15.16:0/3704563034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd81082b0 0x7f4cd819a990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b830 tx=0x7f4cd0004ce0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cdcdb9700 1 -- 172.21.15.16:0/3704563034 reap_dead start 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 shutdown_connections 2024-04-25T01:25:57.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 >> 172.21.15.16:0/3704563034 conn(0x7f4cd80780e0 msgr2=0x7f4cd810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:57.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 shutdown_connections 2024-04-25T01:25:57.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.244+0000 7f4cde81c700 1 -- 172.21.15.16:0/3704563034 wait complete. 2024-04-25T01:25:57.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:57 smithi038 ceph-mon[32841]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T01:25:57.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:57 smithi156 ceph-mon[32631]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T01:25:57.547 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:57 smithi016 ceph-mon[29153]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 -- 172.21.15.16:0/457961389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c1075e0 msgr2=0x7f176c10f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 --2- 172.21.15.16:0/457961389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c1075e0 0x7f176c10f1b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f176400b0d0 tx=0x7f176400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 -- 172.21.15.16:0/457961389 shutdown_connections 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 --2- 172.21.15.16:0/457961389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c1075e0 0x7f176c10f1b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 --2- 172.21.15.16:0/457961389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c102510 0x7f176c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 --2- 172.21.15.16:0/457961389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f176c102ec0 0x7f176c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 -- 172.21.15.16:0/457961389 >> 172.21.15.16:0/457961389 conn(0x7f176c0793e0 msgr2=0x7f176c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 -- 172.21.15.16:0/457961389 shutdown_connections 2024-04-25T01:25:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.673+0000 7f1773cc5700 1 -- 172.21.15.16:0/457961389 wait complete. 2024-04-25T01:25:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 Processor -- start 2024-04-25T01:25:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 -- start start 2024-04-25T01:25:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c102510 0x7f176c1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c102ec0 0x7f176c1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f176c1075e0 0x7f176c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f176c112480 con 0x7f176c102510 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f176c112300 con 0x7f176c102ec0 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1773cc5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f176c112600 con 0x7f176c1075e0 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c102510 0x7f176c1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c102510 0x7f176c1a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59012/0 (socket says 172.21.15.16:59012) 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 -- 172.21.15.16:0/236659873 learned_addr learned my addr 172.21.15.16:0/236659873 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 -- 172.21.15.16:0/236659873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f176c1075e0 msgr2=0x7f176c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f176c1075e0 0x7f176c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 -- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c102ec0 msgr2=0x7f176c1a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f176c102ec0 0x7f176c1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 -- 172.21.15.16:0/236659873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c19b0a0 con 0x7f176c102510 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1771a61700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c102510 0x7f176c1a0780 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f176800c920 tx=0x7f176800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.674+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768007b00 con 0x7f176c102510 2024-04-25T01:25:57.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.675+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176c19b390 con 0x7f176c102510 2024-04-25T01:25:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.675+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1768004ce0 con 0x7f176c102510 2024-04-25T01:25:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.675+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768005560 con 0x7f176c102510 2024-04-25T01:25:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.675+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c1a7650 con 0x7f176c102510 2024-04-25T01:25:57.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.675+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f176c101400 con 0x7f176c102510 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.677+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1768004750 con 0x7f176c102510 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.677+0000 7f1762ffd700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f17580a2cb0 0x7f17580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.677+0000 7f1771260700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f17580a2cb0 0x7f17580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.677+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f17680cf060 con 0x7f176c102510 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.678+0000 7f1762ffd700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] conn(0x7f17580ac490 0x7f17580ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.678+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 --> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17580aef60 con 0x7f17580ac490 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.678+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f17680cf540 con 0x7f176c102510 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.678+0000 7f1771260700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f17580a2cb0 0x7f17580a5170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f175c005fd0 tx=0x7f175c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.678+0000 7f1772262700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] conn(0x7f17580ac490 0x7f17580ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.678+0000 7f1772262700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] conn(0x7f17580ac490 0x7f17580ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.679+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== osd.10 v2:172.21.15.38:6824/3034934986 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f17580aef60 con 0x7f17580ac490 2024-04-25T01:25:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.699+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 --> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f176c1039e0 con 0x7f17580ac490 2024-04-25T01:25:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.699+0000 7f1762ffd700 1 -- 172.21.15.16:0/236659873 <== osd.10 v2:172.21.15.38:6824/3034934986 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f176c1039e0 con 0x7f17580ac490 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] conn(0x7f17580ac490 msgr2=0x7f17580ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] conn(0x7f17580ac490 0x7f17580ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f17580a2cb0 msgr2=0x7f17580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f17580a2cb0 0x7f17580a5170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f175c005fd0 tx=0x7f175c005ee0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c102510 msgr2=0x7f176c1a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 --2- 172.21.15.16:0/236659873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f176c102510 0x7f176c1a0780 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f176800c920 tx=0x7f176800cce0 comp rx=0 tx=0).stop 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1772262700 1 -- 172.21.15.16:0/236659873 reap_dead start 2024-04-25T01:25:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 shutdown_connections 2024-04-25T01:25:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 >> 172.21.15.16:0/236659873 conn(0x7f176c0793e0 msgr2=0x7f176c10d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 shutdown_connections 2024-04-25T01:25:57.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:57.700+0000 7f1773cc5700 1 -- 172.21.15.16:0/236659873 wait complete. 2024-04-25T01:25:58.326 INFO:teuthology.orchestra.run.smithi016.stdout:141733920827 2024-04-25T01:25:58.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T01:25:59.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:25:59 smithi016 ceph-mon[29153]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:25:59.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:25:59 smithi038 ceph-mon[32841]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:25:59.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:25:59 smithi156 ceph-mon[32631]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:25:59.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 -- 172.21.15.16:0/2213682708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86041013c0 msgr2=0x7f86041017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:59.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 --2- 172.21.15.16:0/2213682708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86041013c0 0x7f86041017a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f85f40099c0 tx=0x7f85f4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 -- 172.21.15.16:0/2213682708 shutdown_connections 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 --2- 172.21.15.16:0/2213682708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8604106450 0x7f860410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 --2- 172.21.15.16:0/2213682708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86041013c0 0x7f86041017a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 --2- 172.21.15.16:0/2213682708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8604101d70 0x7f8604105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 -- 172.21.15.16:0/2213682708 >> 172.21.15.16:0/2213682708 conn(0x7f86040780e0 msgr2=0x7f86040784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 -- 172.21.15.16:0/2213682708 shutdown_connections 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 -- 172.21.15.16:0/2213682708 wait complete. 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 Processor -- start 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.453+0000 7f860aea6700 1 -- start start 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f860aea6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86041013c0 0x7f860419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f860aea6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8604101d70 0x7f860419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f860aea6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604106450 0x7f8604197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f860aea6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f86041113d0 con 0x7f8604106450 2024-04-25T01:25:59.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f860aea6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8604111250 con 0x7f8604101d70 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f860aea6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8604111550 con 0x7f86041013c0 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604106450 0x7f8604197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604106450 0x7f8604197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59028/0 (socket says 172.21.15.16:59028) 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 -- 172.21.15.16:0/2705529701 learned_addr learned my addr 172.21.15.16:0/2705529701 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8603fff700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8604101d70 0x7f860419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 -- 172.21.15.16:0/2705529701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86041013c0 msgr2=0x7f860419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8608c42700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86041013c0 0x7f860419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86041013c0 0x7f860419d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 -- 172.21.15.16:0/2705529701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8604101d70 msgr2=0x7f860419dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8604101d70 0x7f860419dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 -- 172.21.15.16:0/2705529701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8604198010 con 0x7f8604106450 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8603fff700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8604101d70 0x7f860419dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8608c42700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86041013c0 0x7f860419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8609443700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604106450 0x7f8604197730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00b970 tx=0x7f85fc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.454+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc019070 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.455+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85fc014440 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.455+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8604198300 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.455+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00ce60 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.455+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86041a45e0 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.456+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8604062980 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.457+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f85fc01c070 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.457+0000 7f8601ffb700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85ec0a2cb0 0x7f85ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.457+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f85fc0ce3a0 con 0x7f8604106450 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8601ffb700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] conn(0x7f85ec0ac490 0x7f85ec0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 --> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85ec0aef60 con 0x7f85ec0ac490 2024-04-25T01:25:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f85fc0d5050 con 0x7f8604106450 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8608c42700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85ec0a2cb0 0x7f85ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8603fff700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] conn(0x7f85ec0ac490 0x7f85ec0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8608c42700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85ec0a2cb0 0x7f85ec0a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006010 tx=0x7f85f4005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.458+0000 7f8603fff700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] conn(0x7f85ec0ac490 0x7f85ec0ae8b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.459+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== osd.9 v2:172.21.15.16:6826/2042480684 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f85ec0aef60 con 0x7f85ec0ac490 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 --> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8604062e00 con 0x7f85ec0ac490 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f8601ffb700 1 -- 172.21.15.16:0/2705529701 <== osd.9 v2:172.21.15.16:6826/2042480684 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8604062e00 con 0x7f85ec0ac490 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] conn(0x7f85ec0ac490 msgr2=0x7f85ec0ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:59.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] conn(0x7f85ec0ac490 0x7f85ec0ae8b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85ec0a2cb0 msgr2=0x7f85ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85ec0a2cb0 0x7f85ec0a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006010 tx=0x7f85f4005e80 comp rx=0 tx=0).stop 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604106450 msgr2=0x7f8604197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 --2- 172.21.15.16:0/2705529701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604106450 0x7f8604197730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00b970 tx=0x7f85fc00bd30 comp rx=0 tx=0).stop 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f8609443700 1 -- 172.21.15.16:0/2705529701 reap_dead start 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 shutdown_connections 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.476+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 >> 172.21.15.16:0/2705529701 conn(0x7f86040780e0 msgr2=0x7f86040ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.477+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 shutdown_connections 2024-04-25T01:25:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:25:59.477+0000 7f860aea6700 1 -- 172.21.15.16:0/2705529701 wait complete. 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-25T01:26:00.446 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-25T01:26:00.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-04-25T01:26:00.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-25T01:26:00.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-04-25T01:26:01.406 INFO:teuthology.orchestra.run.smithi016.stdout:309237645339 2024-04-25T01:26:01.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T01:26:01.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:26:01.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:26:01.429 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-04-25T01:26:01.430 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:01 smithi016 ceph-mon[29153]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:26:01.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:26:01.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:26:01.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-25T01:26:01.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:26:01.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-04-25T01:26:01.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:01 smithi038 ceph-mon[32841]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:26:01.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-04-25T01:26:01.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:01 smithi156 ceph-mon[32631]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 -- 172.21.15.16:0/3024330447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33801082b0 msgr2=0x7f3380108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 --2- 172.21.15.16:0/3024330447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33801082b0 0x7f3380108690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f337c009990 tx=0x7f337c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 -- 172.21.15.16:0/3024330447 shutdown_connections 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 --2- 172.21.15.16:0/3024330447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338006a8e0 0x7f338010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 --2- 172.21.15.16:0/3024330447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33801082b0 0x7f3380108690 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 --2- 172.21.15.16:0/3024330447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3380069f40 0x7f338006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 -- 172.21.15.16:0/3024330447 >> 172.21.15.16:0/3024330447 conn(0x7f33800780e0 msgr2=0x7f33800784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 -- 172.21.15.16:0/3024330447 shutdown_connections 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 -- 172.21.15.16:0/3024330447 wait complete. 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 Processor -- start 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.624+0000 7f33876d0700 1 -- start start 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33876d0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3380069f40 0x7f338019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33876d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f338006a8e0 0x7f338019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33876d0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33801082b0 0x7f33801987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33876d0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3380112410 con 0x7f3380069f40 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33876d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3380112290 con 0x7f338006a8e0 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33876d0700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3380112590 con 0x7f33801082b0 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3380069f40 0x7f338019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3380069f40 0x7f338019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59040/0 (socket says 172.21.15.16:59040) 2024-04-25T01:26:01.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 -- 172.21.15.16:0/1208749613 learned_addr learned my addr 172.21.15.16:0/1208749613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f3385c6d700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33801082b0 0x7f33801987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f3384c6b700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f338006a8e0 0x7f338019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 -- 172.21.15.16:0/1208749613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33801082b0 msgr2=0x7f33801987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33801082b0 0x7f33801987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 -- 172.21.15.16:0/1208749613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f338006a8e0 msgr2=0x7f338019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f338006a8e0 0x7f338019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 -- 172.21.15.16:0/1208749613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3380198fb0 con 0x7f3380069f40 2024-04-25T01:26:01.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f3384c6b700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f338006a8e0 0x7f338019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f338546c700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3380069f40 0x7f338019e650 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f337c005910 tx=0x7f337c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c027070 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.625+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f337c003f90 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.626+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3380199240 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.626+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c022d70 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.626+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33801a54b0 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.626+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3380062940 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.628+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f337c02a070 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.628+0000 7f33767fc700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f336c0a2cb0 0x7f336c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.628+0000 7f3384c6b700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f336c0a2cb0 0x7f336c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.628+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f337c0d0760 con 0x7f3380069f40 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.629+0000 7f33767fc700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] conn(0x7f336c0ac8f0 0x7f336c0aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.629+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 --> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f337c001d30 con 0x7f336c0ac8f0 2024-04-25T01:26:01.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.629+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f337c0d0bb0 con 0x7f3380069f40 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.629+0000 7f3385c6d700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] conn(0x7f336c0ac8f0 0x7f336c0aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.629+0000 7f3384c6b700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f336c0a2cb0 0x7f336c0a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3370005fd0 tx=0x7f3370005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.629+0000 7f3385c6d700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] conn(0x7f336c0ac8f0 0x7f336c0aed90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.630+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== osd.3 v2:172.21.15.16:6810/3649873026 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f337c001d30 con 0x7f336c0ac8f0 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.646+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 --> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3380062dc0 con 0x7f336c0ac8f0 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.646+0000 7f33767fc700 1 -- 172.21.15.16:0/1208749613 <== osd.3 v2:172.21.15.16:6810/3649873026 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3380062dc0 con 0x7f336c0ac8f0 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] conn(0x7f336c0ac8f0 msgr2=0x7f336c0aed90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] conn(0x7f336c0ac8f0 0x7f336c0aed90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f336c0a2cb0 msgr2=0x7f336c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f336c0a2cb0 0x7f336c0a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3370005fd0 tx=0x7f3370005ee0 comp rx=0 tx=0).stop 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3380069f40 msgr2=0x7f338019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:01.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 --2- 172.21.15.16:0/1208749613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3380069f40 0x7f338019e650 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f337c005910 tx=0x7f337c009f90 comp rx=0 tx=0).stop 2024-04-25T01:26:01.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f3385c6d700 1 -- 172.21.15.16:0/1208749613 reap_dead start 2024-04-25T01:26:01.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 shutdown_connections 2024-04-25T01:26:01.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 >> 172.21.15.16:0/1208749613 conn(0x7f33800780e0 msgr2=0x7f33800fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:01.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 shutdown_connections 2024-04-25T01:26:01.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:01.647+0000 7f33876d0700 1 -- 172.21.15.16:0/1208749613 wait complete. 2024-04-25T01:26:02.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:02 smithi016 ceph-mon[29153]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:26:02.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:02 smithi038 ceph-mon[32841]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:26:02.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:02 smithi156 ceph-mon[32631]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:26:02.638 INFO:teuthology.orchestra.run.smithi016.stdout:163208757301 2024-04-25T01:26:02.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T01:26:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 -- 172.21.15.16:0/349690023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16141064c0 msgr2=0x7f161410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 --2- 172.21.15.16:0/349690023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16141064c0 0x7f161410e060 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f160c00b0d0 tx=0x7f160c00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 -- 172.21.15.16:0/349690023 shutdown_connections 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 --2- 172.21.15.16:0/349690023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16141064c0 0x7f161410e060 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 --2- 172.21.15.16:0/349690023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1614101770 0x7f1614101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 --2- 172.21.15.16:0/349690023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1614102120 0x7f1614105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 -- 172.21.15.16:0/349690023 >> 172.21.15.16:0/349690023 conn(0x7f16140780e0 msgr2=0x7f16140784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 -- 172.21.15.16:0/349690023 shutdown_connections 2024-04-25T01:26:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.868+0000 7f161afaa700 1 -- 172.21.15.16:0/349690023 wait complete. 2024-04-25T01:26:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 Processor -- start 2024-04-25T01:26:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 -- start start 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1614101770 0x7f16141997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1614102120 0x7f1614199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16141064c0 0x7f16141a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f16141113b0 con 0x7f1614102120 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1614111230 con 0x7f16141064c0 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f161afaa700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1614111530 con 0x7f1614101770 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f1613fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1614102120 0x7f1614199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f1613fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1614102120 0x7f1614199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53114/0 (socket says 172.21.15.16:53114) 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f1613fff700 1 -- 172.21.15.16:0/3740268538 learned_addr learned my addr 172.21.15.16:0/3740268538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.869+0000 7f1618d46700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1614101770 0x7f16141997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1619547700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16141064c0 0x7f16141a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1613fff700 1 -- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1614101770 msgr2=0x7f16141997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1613fff700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1614101770 0x7f16141997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1613fff700 1 -- 172.21.15.16:0/3740268538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16141064c0 msgr2=0x7f16141a4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1613fff700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16141064c0 0x7f16141a4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1613fff700 1 -- 172.21.15.16:0/3740268538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16141a4d90 con 0x7f1614102120 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1618d46700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1614101770 0x7f16141997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1613fff700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1614102120 0x7f1614199d10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f160000d950 tx=0x7f160000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600017070 con 0x7f1614102120 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f160000fd40 con 0x7f1614102120 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1614197620 con 0x7f1614102120 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.870+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160000e3b0 con 0x7f1614102120 2024-04-25T01:26:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.871+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1614197b60 con 0x7f1614102120 2024-04-25T01:26:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.871+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1614062900 con 0x7f1614102120 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.873+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f160001a070 con 0x7f1614102120 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.873+0000 7f1611ffb700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f16040a2cb0 0x7f16040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1618d46700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f16040a2cb0 0x7f16040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f16000cedd0 con 0x7f1614102120 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1611ffb700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] conn(0x7f16040ac8f0 0x7f16040aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 --> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1600025ea0 con 0x7f16040ac8f0 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f16000cf220 con 0x7f1614102120 2024-04-25T01:26:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1618d46700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f16040a2cb0 0x7f16040a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1608009de0 tx=0x7f1608009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.874+0000 7f1619547700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] conn(0x7f16040ac8f0 0x7f16040aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.875+0000 7f1619547700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] conn(0x7f16040ac8f0 0x7f16040aed90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.876+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== osd.8 v2:172.21.15.156:6816/1052118798 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1600025ea0 con 0x7f16040ac8f0 2024-04-25T01:26:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.898+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 --> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1614062d80 con 0x7f16040ac8f0 2024-04-25T01:26:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f1611ffb700 1 -- 172.21.15.16:0/3740268538 <== osd.8 v2:172.21.15.156:6816/1052118798 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1614062d80 con 0x7f16040ac8f0 2024-04-25T01:26:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] conn(0x7f16040ac8f0 msgr2=0x7f16040aed90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] conn(0x7f16040ac8f0 0x7f16040aed90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f16040a2cb0 msgr2=0x7f16040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f16040a2cb0 0x7f16040a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1608009de0 tx=0x7f1608009190 comp rx=0 tx=0).stop 2024-04-25T01:26:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1614102120 msgr2=0x7f1614199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 --2- 172.21.15.16:0/3740268538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1614102120 0x7f1614199d10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f160000d950 tx=0x7f160000dc60 comp rx=0 tx=0).stop 2024-04-25T01:26:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f1619547700 1 -- 172.21.15.16:0/3740268538 reap_dead start 2024-04-25T01:26:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 shutdown_connections 2024-04-25T01:26:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 >> 172.21.15.16:0/3740268538 conn(0x7f16140780e0 msgr2=0x7f1614100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 shutdown_connections 2024-04-25T01:26:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:02.899+0000 7f161afaa700 1 -- 172.21.15.16:0/3740268538 wait complete. 2024-04-25T01:26:03.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:03 smithi038 ceph-mon[32841]: pgmap v318: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:26:03.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:03 smithi156 ceph-mon[32631]: pgmap v318: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:26:03.522 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:03 smithi016 ceph-mon[29153]: pgmap v318: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:26:03.551 INFO:teuthology.orchestra.run.smithi016.stdout:107374182464 2024-04-25T01:26:03.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 -- 172.21.15.16:0/1253770127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570101800 msgr2=0x7f3570106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 --2- 172.21.15.16:0/1253770127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570101800 0x7f3570106f40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f35580099c0 tx=0x7f3558009cd0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 -- 172.21.15.16:0/1253770127 shutdown_connections 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 --2- 172.21.15.16:0/1253770127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3570107670 0x7f357010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 --2- 172.21.15.16:0/1253770127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3570100e50 0x7f3570101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 --2- 172.21.15.16:0/1253770127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570101800 0x7f3570106f40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.777+0000 7f3575346700 1 -- 172.21.15.16:0/1253770127 >> 172.21.15.16:0/1253770127 conn(0x7f3570076450 msgr2=0x7f3570076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.778+0000 7f3575346700 1 -- 172.21.15.16:0/1253770127 shutdown_connections 2024-04-25T01:26:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.778+0000 7f3575346700 1 -- 172.21.15.16:0/1253770127 wait complete. 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.778+0000 7f3575346700 1 Processor -- start 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.778+0000 7f3575346700 1 -- start start 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3570100e50 0x7f357019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3570101800 0x7f357019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570107670 0x7f35701985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3570112430 con 0x7f3570107670 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35701122b0 con 0x7f3570100e50 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35701125b0 con 0x7f3570101800 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570107670 0x7f35701985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570107670 0x7f35701985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53124/0 (socket says 172.21.15.16:53124) 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 -- 172.21.15.16:0/3132500433 learned_addr learned my addr 172.21.15.16:0/3132500433 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356e7fc700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3570101800 0x7f357019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356effd700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3570100e50 0x7f357019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 -- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3570101800 msgr2=0x7f357019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3570101800 0x7f357019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 -- 172.21.15.16:0/3132500433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3570100e50 msgr2=0x7f357019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3570100e50 0x7f357019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 -- 172.21.15.16:0/3132500433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3570198d50 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f356f7fe700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570107670 0x7f35701985f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3568007ee0 tx=0x7f356800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800dbd0 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3570199040 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.779+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f356800dd70 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.780+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568005990 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.780+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35701a5300 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.780+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35700629c0 con 0x7f3570107670 2024-04-25T01:26:03.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f356801a020 con 0x7f3570107670 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f3567fff700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f355c0a2cb0 0x7f355c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f356effd700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f355c0a2cb0 0x7f355c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f35680cf350 con 0x7f3570107670 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f3567fff700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] conn(0x7f355c0ac8f0 0x7f355c0aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 --> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3568023e70 con 0x7f355c0ac8f0 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.783+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f35680cf7a0 con 0x7f3570107670 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.784+0000 7f356effd700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f355c0a2cb0 0x7f355c0a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3560005ea0 tx=0x7f3560005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.784+0000 7f356e7fc700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] conn(0x7f355c0ac8f0 0x7f355c0aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.784+0000 7f356e7fc700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] conn(0x7f355c0ac8f0 0x7f355c0aed90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:03.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.785+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== osd.11 v2:172.21.15.156:6824/1283302638 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3568023e70 con 0x7f355c0ac8f0 2024-04-25T01:26:03.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.802+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 --> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3570062dc0 con 0x7f355c0ac8f0 2024-04-25T01:26:03.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.802+0000 7f3567fff700 1 -- 172.21.15.16:0/3132500433 <== osd.11 v2:172.21.15.156:6824/1283302638 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3570062dc0 con 0x7f355c0ac8f0 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] conn(0x7f355c0ac8f0 msgr2=0x7f355c0aed90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] conn(0x7f355c0ac8f0 0x7f355c0aed90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f355c0a2cb0 msgr2=0x7f355c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f355c0a2cb0 0x7f355c0a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3560005ea0 tx=0x7f3560005e30 comp rx=0 tx=0).stop 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570107670 msgr2=0x7f35701985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 --2- 172.21.15.16:0/3132500433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3570107670 0x7f35701985f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3568007ee0 tx=0x7f356800d3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f356f7fe700 1 -- 172.21.15.16:0/3132500433 reap_dead start 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 shutdown_connections 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 >> 172.21.15.16:0/3132500433 conn(0x7f3570076450 msgr2=0x7f35700ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 shutdown_connections 2024-04-25T01:26:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:03.803+0000 7f3575346700 1 -- 172.21.15.16:0/3132500433 wait complete. 2024-04-25T01:26:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 -- 172.21.15.16:0/3725750511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd001075a0 msgr2=0x7fcd0010f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 --2- 172.21.15.16:0/3725750511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd001075a0 0x7fcd0010f170 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fccf800b0a0 tx=0x7fccf800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 -- 172.21.15.16:0/3725750511 shutdown_connections 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 --2- 172.21.15.16:0/3725750511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd001075a0 0x7fcd0010f170 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 --2- 172.21.15.16:0/3725750511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd001024e0 0x7fcd001028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 --2- 172.21.15.16:0/3725750511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd00102e90 0x7fcd00106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 -- 172.21.15.16:0/3725750511 >> 172.21.15.16:0/3725750511 conn(0x7fcd000793e0 msgr2=0x7fcd000797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 -- 172.21.15.16:0/3725750511 shutdown_connections 2024-04-25T01:26:04.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.035+0000 7fcd074ee700 1 -- 172.21.15.16:0/3725750511 wait complete. 2024-04-25T01:26:04.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 Processor -- start 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 -- start start 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd001024e0 0x7fcd0019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd00102e90 0x7fcd0019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd001075a0 0x7fcd001986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd00112430 con 0x7fcd00102e90 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcd001122b0 con 0x7fcd001024e0 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd074ee700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcd001125b0 con 0x7fcd001075a0 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd00102e90 0x7fcd0019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd00102e90 0x7fcd0019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53144/0 (socket says 172.21.15.16:53144) 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 -- 172.21.15.16:0/1732657108 learned_addr learned my addr 172.21.15.16:0/1732657108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd0528a700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd001024e0 0x7fcd0019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd05a8b700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd001075a0 0x7fcd001986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 -- 172.21.15.16:0/1732657108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd001075a0 msgr2=0x7fcd001986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd001075a0 0x7fcd001986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 -- 172.21.15.16:0/1732657108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd001024e0 msgr2=0x7fcd0019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd001024e0 0x7fcd0019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.036+0000 7fcd04a89700 1 -- 172.21.15.16:0/1732657108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd00198fb0 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.037+0000 7fcd04a89700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd00102e90 0x7fcd0019eb40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00d980 tx=0x7fccfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.037+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc017070 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.037+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd001992a0 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.037+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccfc00fd40 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.037+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc00e3b0 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.037+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd001a5580 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.038+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcd00062980 con 0x7fcd00102e90 2024-04-25T01:26:04.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.039+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fccfc00f890 con 0x7fcd00102e90 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fccf27fc700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fccec0a2cb0 0x7fccec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fcd0528a700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fccec0a2cb0 0x7fccec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fccfc0cefb0 con 0x7fcd00102e90 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fccf27fc700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] conn(0x7fccec0ac8f0 0x7fccec0aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 --> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccfc005690 con 0x7fccec0ac8f0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fccfc0cf400 con 0x7fcd00102e90 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fcd05a8b700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] conn(0x7fccec0ac8f0 0x7fccec0aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.040+0000 7fcd0528a700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fccec0a2cb0 0x7fccec0a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fccf4005950 tx=0x7fccf400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.041+0000 7fcd05a8b700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] conn(0x7fccec0ac8f0 0x7fccec0aed90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.041+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== osd.0 v2:172.21.15.16:6802/1017436960 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fccfc005690 con 0x7fccec0ac8f0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.058+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 --> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcd00062d80 con 0x7fccec0ac8f0 2024-04-25T01:26:04.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.058+0000 7fccf27fc700 1 -- 172.21.15.16:0/1732657108 <== osd.0 v2:172.21.15.16:6802/1017436960 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcd00062d80 con 0x7fccec0ac8f0 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] conn(0x7fccec0ac8f0 msgr2=0x7fccec0aed90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] conn(0x7fccec0ac8f0 0x7fccec0aed90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fccec0a2cb0 msgr2=0x7fccec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fccec0a2cb0 0x7fccec0a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fccf4005950 tx=0x7fccf400b410 comp rx=0 tx=0).stop 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd00102e90 msgr2=0x7fcd0019eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 --2- 172.21.15.16:0/1732657108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd00102e90 0x7fcd0019eb40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00d980 tx=0x7fccfc00dc90 comp rx=0 tx=0).stop 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd05a8b700 1 -- 172.21.15.16:0/1732657108 reap_dead start 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 shutdown_connections 2024-04-25T01:26:04.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 >> 172.21.15.16:0/1732657108 conn(0x7fcd000793e0 msgr2=0x7fcd00100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:04.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 shutdown_connections 2024-04-25T01:26:04.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:04.059+0000 7fcd074ee700 1 -- 172.21.15.16:0/1732657108 wait complete. 2024-04-25T01:26:04.697 INFO:teuthology.orchestra.run.smithi016.stdout:253403070503 2024-04-25T01:26:04.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T01:26:05.337 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:05 smithi016 ceph-mon[29153]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-25T01:26:05.369 INFO:teuthology.orchestra.run.smithi016.stdout:34359738449 2024-04-25T01:26:05.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T01:26:05.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:05 smithi038 ceph-mon[32841]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-25T01:26:05.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:05 smithi156 ceph-mon[32631]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-25T01:26:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:05.854 INFO:teuthology.orchestra.run.smithi016.stdout:287762808866 2024-04-25T01:26:05.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T01:26:06.509 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:06.522 INFO:teuthology.orchestra.run.smithi016.stdout:343597383704 2024-04-25T01:26:06.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T01:26:06.569 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:07.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:07 smithi038 ceph-mon[32841]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:26:07.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:07 smithi156 ceph-mon[32631]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:26:07.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:07 smithi016 ceph-mon[29153]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:26:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:09.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:09 smithi016 ceph-mon[29153]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:26:09.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:09 smithi038 ceph-mon[32841]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:26:09.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:09 smithi156 ceph-mon[32631]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:26:11.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:11 smithi038 ceph-mon[32841]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s; 6.3 KiB/s, 1 objects/s recovering 2024-04-25T01:26:11.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:11 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3982199254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:26:11.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:11 smithi156 ceph-mon[32631]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s; 6.3 KiB/s, 1 objects/s recovering 2024-04-25T01:26:11.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:11 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3982199254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:26:11.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:11 smithi016 ceph-mon[29153]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s; 6.3 KiB/s, 1 objects/s recovering 2024-04-25T01:26:11.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:11 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3982199254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:26:13.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:13 smithi016 ceph-mon[29153]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 5.7 KiB/s, 1 objects/s recovering 2024-04-25T01:26:13.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 -- 172.21.15.16:0/947076741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 msgr2=0x7f114c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:13.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 --2- 172.21.15.16:0/947076741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c107580 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f113c009990 tx=0x7f113c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 -- 172.21.15.16:0/947076741 shutdown_connections 2024-04-25T01:26:13.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 --2- 172.21.15.16:0/947076741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f114c101af0 0x7f114c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 --2- 172.21.15.16:0/947076741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c107580 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 --2- 172.21.15.16:0/947076741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 0x7f114c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 -- 172.21.15.16:0/947076741 >> 172.21.15.16:0/947076741 conn(0x7f114c0780e0 msgr2=0x7f114c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 -- 172.21.15.16:0/947076741 shutdown_connections 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 -- 172.21.15.16:0/947076741 wait complete. 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 Processor -- start 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.709+0000 7f1152612700 1 -- start start 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1152612700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 0x7f114c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1152612700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f114c101af0 0x7f114c1036d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1152612700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c103c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1152612700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1112c0 con 0x7f114c1071a0 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1152612700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111140 con 0x7f114c101af0 2024-04-25T01:26:13.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1152612700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111440 con 0x7f114c101130 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stdout:309237645341 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c103c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c103c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53164/0 (socket says 172.21.15.16:53164) 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 -- 172.21.15.16:0/3982199254 learned_addr learned my addr 172.21.15.16:0/3982199254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f114b7fe700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f114c101af0 0x7f114c1036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f114bfff700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 0x7f114c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 -- 172.21.15.16:0/3982199254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 msgr2=0x7f114c103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 0x7f114c103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 -- 172.21.15.16:0/3982199254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f114c101af0 msgr2=0x7f114c1036d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f114c101af0 0x7f114c1036d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f1150baf700 1 -- 172.21.15.16:0/3982199254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c1a6df0 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.710+0000 7f114bfff700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 0x7f114c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.711+0000 7f1150baf700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c103c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1144007d70 tx=0x7f114400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.711+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144016020 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.711+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c1a70e0 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.711+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f114400ad40 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.711+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11440115e0 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.711+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c1a75d0 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.712+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f114c068f20 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.713+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1144019020 con 0x7f114c1071a0 2024-04-25T01:26:13.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.714+0000 7f11497fa700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11340a2cb0 0x7f11340a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.714+0000 7f114bfff700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11340a2cb0 0x7f11340a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.714+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f11440cefc0 con 0x7f114c1071a0 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.715+0000 7f114bfff700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11340a2cb0 0x7f11340a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f113c006010 tx=0x7f113c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.719+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1144020020 con 0x7f114c1071a0 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.926+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f114c1a7960 con 0x7f114c1071a0 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.926+0000 7f11497fa700 1 -- 172.21.15.16:0/3982199254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f114409bf50 con 0x7f114c1071a0 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.928+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11340a2cb0 msgr2=0x7f11340a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.928+0000 7f1152612700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11340a2cb0 0x7f11340a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f113c006010 tx=0x7f113c00b540 comp rx=0 tx=0).stop 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.928+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 msgr2=0x7f114c103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c103c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1144007d70 tx=0x7f114400a340 comp rx=0 tx=0).stop 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 shutdown_connections 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11340a2cb0 0x7f11340a5170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f114c1071a0 0x7f114c103c10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f114c101130 0x7f114c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 --2- 172.21.15.16:0/3982199254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f114c101af0 0x7f114c1036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 >> 172.21.15.16:0/3982199254 conn(0x7f114c0780e0 msgr2=0x7f114c105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 shutdown_connections 2024-04-25T01:26:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:10.929+0000 7f1152612700 1 -- 172.21.15.16:0/3982199254 wait complete. 2024-04-25T01:26:13.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:13 smithi038 ceph-mon[32841]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 5.7 KiB/s, 1 objects/s recovering 2024-04-25T01:26:13.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:13 smithi156 ceph-mon[32631]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 5.7 KiB/s, 1 objects/s recovering 2024-04-25T01:26:14.080 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:14.122 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645339 got 309237645341 for osd.10 2024-04-25T01:26:14.122 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:15.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:15 smithi016 ceph-mon[29153]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 5.0 KiB/s, 1 objects/s recovering 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 -- 172.21.15.16:0/3342332752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 msgr2=0x7fa83410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 --2- 172.21.15.16:0/3342332752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa83410dfd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00b0a0 tx=0x7fa82c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 -- 172.21.15.16:0/3342332752 shutdown_connections 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 --2- 172.21.15.16:0/3342332752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa83410dfd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 --2- 172.21.15.16:0/3342332752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834107190 0x7fa834107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 --2- 172.21.15.16:0/3342332752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834101120 0x7fa8341015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 -- 172.21.15.16:0/3342332752 >> 172.21.15.16:0/3342332752 conn(0x7fa8340780a0 msgr2=0x7fa8340784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:15.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 -- 172.21.15.16:0/3342332752 shutdown_connections 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.345+0000 7fa83893e700 1 -- 172.21.15.16:0/3342332752 wait complete. 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 Processor -- start 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 -- start start 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834101120 0x7fa8341996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa834199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834107190 0x7fa8341a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8341112d0 con 0x7fa834101ae0 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa834111150 con 0x7fa834107190 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.346+0000 7fa83893e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa834111450 con 0x7fa834101120 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa834199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa83259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834101120 0x7fa8341996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa832d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834107190 0x7fa8341a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa834199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43388/0 (socket says 172.21.15.16:43388) 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 -- 172.21.15.16:0/2962448502 learned_addr learned my addr 172.21.15.16:0/2962448502 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 -- 172.21.15.16:0/2962448502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834101120 msgr2=0x7fa8341996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834101120 0x7fa8341996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 -- 172.21.15.16:0/2962448502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834107190 msgr2=0x7fa8341a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834107190 0x7fa8341a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 -- 172.21.15.16:0/2962448502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8341a4c70 con 0x7fa834101ae0 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa832d9d700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834107190 0x7fa8341a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa83259c700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834101120 0x7fa8341996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa831d9b700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa834199bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00e9a0 tx=0x7fa81c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00ca10 con 0x7fa834101ae0 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa81c00cbb0 con 0x7fa834101ae0 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8341a4ea0 con 0x7fa834101ae0 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c018680 con 0x7fa834101ae0 2024-04-25T01:26:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.347+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8341a5350 con 0x7fa834101ae0 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.349+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa834068f20 con 0x7fa834101ae0 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.349+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa81c010430 con 0x7fa834101ae0 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.350+0000 7fa82b7fe700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8200a2cb0 0x7fa8200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.350+0000 7fa83259c700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8200a2cb0 0x7fa8200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.350+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa81c0cf330 con 0x7fa834101ae0 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.351+0000 7fa83259c700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8200a2cb0 0x7fa8200a5170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa824009fd0 tx=0x7fa824009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:15.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.355+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa81c09c110 con 0x7fa834101ae0 2024-04-25T01:26:15.465 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:15 smithi038 ceph-mon[32841]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 5.0 KiB/s, 1 objects/s recovering 2024-04-25T01:26:15.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:15 smithi156 ceph-mon[32631]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 5.0 KiB/s, 1 objects/s recovering 2024-04-25T01:26:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.565+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa83402db30 con 0x7fa834101ae0 2024-04-25T01:26:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.566+0000 7fa82b7fe700 1 -- 172.21.15.16:0/2962448502 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa81c09ba10 con 0x7fa834101ae0 2024-04-25T01:26:15.545 INFO:teuthology.orchestra.run.smithi016.stdout:107374182467 2024-04-25T01:26:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8200a2cb0 msgr2=0x7fa8200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8200a2cb0 0x7fa8200a5170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa824009fd0 tx=0x7fa824009260 comp rx=0 tx=0).stop 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 msgr2=0x7fa834199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa834199bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00e9a0 tx=0x7fa81c00ecb0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 shutdown_connections 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8200a2cb0 0x7fa8200a5170 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa834107190 0x7fa8341a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa834101120 0x7fa8341996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 --2- 172.21.15.16:0/2962448502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa834101ae0 0x7fa834199bf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 >> 172.21.15.16:0/2962448502 conn(0x7fa8340780a0 msgr2=0x7fa8340ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 shutdown_connections 2024-04-25T01:26:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:15.568+0000 7fa83893e700 1 -- 172.21.15.16:0/2962448502 wait complete. 2024-04-25T01:26:16.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:16 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:16.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:16 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2962448502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:26:16.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182467 for osd.3 2024-04-25T01:26:16.466 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:16.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:16 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:16.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:16 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2962448502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:26:16.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:16 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:16.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:16 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2962448502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:26:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 -- 172.21.15.16:0/977625750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 msgr2=0x7f98cc10f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 --2- 172.21.15.16:0/977625750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc10f140 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f98c400b0a0 tx=0x7f98c400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 -- 172.21.15.16:0/977625750 shutdown_connections 2024-04-25T01:26:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 --2- 172.21.15.16:0/977625750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc10f140 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 --2- 172.21.15.16:0/977625750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc1024e0 0x7f98cc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:16.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 --2- 172.21.15.16:0/977625750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc102e90 0x7f98cc106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 -- 172.21.15.16:0/977625750 >> 172.21.15.16:0/977625750 conn(0x7f98cc079390 msgr2=0x7f98cc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 -- 172.21.15.16:0/977625750 shutdown_connections 2024-04-25T01:26:16.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.796+0000 7f98d21e4700 1 -- 172.21.15.16:0/977625750 wait complete. 2024-04-25T01:26:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 Processor -- start 2024-04-25T01:26:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 -- start start 2024-04-25T01:26:16.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc1024e0 0x7f98cc074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc102e90 0x7f98cc074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc112400 con 0x7f98cc107570 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc112280 con 0x7f98cc102e90 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.797+0000 7f98d21e4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc112580 con 0x7f98cc1024e0 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43412/0 (socket says 172.21.15.16:43412) 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 -- 172.21.15.16:0/3699744143 learned_addr learned my addr 172.21.15.16:0/3699744143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98caffd700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc102e90 0x7f98cc074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cb7fe700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc1024e0 0x7f98cc074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 -- 172.21.15.16:0/3699744143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc1024e0 msgr2=0x7f98cc074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc1024e0 0x7f98cc074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:16.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 -- 172.21.15.16:0/3699744143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc102e90 msgr2=0x7f98cc074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc102e90 0x7f98cc074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 -- 172.21.15.16:0/3699744143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98cc1a6e20 con 0x7f98cc107570 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98caffd700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc102e90 0x7f98cc074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cb7fe700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc1024e0 0x7f98cc074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98cbfff700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc074fb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f98c400b780 tx=0x7f98c4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4021070 con 0x7f98cc107570 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98c4007e30 con 0x7f98cc107570 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98cc1a70b0 con 0x7f98cc107570 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4004700 con 0x7f98cc107570 2024-04-25T01:26:16.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.798+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98cc1a7510 con 0x7f98cc107570 2024-04-25T01:26:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.799+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98cc068f20 con 0x7f98cc107570 2024-04-25T01:26:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.801+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f98c4007980 con 0x7f98cc107570 2024-04-25T01:26:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.801+0000 7f98c8ff9700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f98b40a2cb0 0x7f98b40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.801+0000 7f98cb7fe700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f98b40a2cb0 0x7f98b40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.801+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f98c40d0050 con 0x7f98cc107570 2024-04-25T01:26:16.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.802+0000 7f98cb7fe700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f98b40a2cb0 0x7f98b40a5170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f98bc011490 tx=0x7f98bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:16.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:16.805+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98c4003c00 con 0x7f98cc107570 2024-04-25T01:26:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.021+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f98cc02db30 con 0x7f98cc107570 2024-04-25T01:26:17.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.022+0000 7f98c8ff9700 1 -- 172.21.15.16:0/3699744143 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f98c409cec0 con 0x7f98cc107570 2024-04-25T01:26:17.002 INFO:teuthology.orchestra.run.smithi016.stdout:141733920831 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f98b40a2cb0 msgr2=0x7f98b40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f98b40a2cb0 0x7f98b40a5170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f98bc011490 tx=0x7f98bc009260 comp rx=0 tx=0).stop 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 msgr2=0x7f98cc074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc074fb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f98c400b780 tx=0x7f98c4004d00 comp rx=0 tx=0).stop 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 shutdown_connections 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f98b40a2cb0 0x7f98b40a5170 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98cc107570 0x7f98cc074fb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98cc1024e0 0x7f98cc074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 --2- 172.21.15.16:0/3699744143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98cc102e90 0x7f98cc074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 >> 172.21.15.16:0/3699744143 conn(0x7f98cc079390 msgr2=0x7f98cc1086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 shutdown_connections 2024-04-25T01:26:17.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:17.024+0000 7f98d21e4700 1 -- 172.21.15.16:0/3699744143 wait complete. 2024-04-25T01:26:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:17 smithi016 ceph-mon[29153]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-04-25T01:26:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:17 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:17 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3699744143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:26:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:17 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:17 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.465 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:17 smithi038 ceph-mon[32841]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-04-25T01:26:17.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:17 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:17 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3699744143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:26:17.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:17 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:17 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:17 smithi156 ceph-mon[32631]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-04-25T01:26:17.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:17 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:17 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3699744143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:26:17.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:17 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.501 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:17 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:17.984 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:18.299 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:18.301 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:18.338 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920831 for osd.4 2024-04-25T01:26:18.339 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:18.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 -- 172.21.15.16:0/2364246547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 msgr2=0x7fc25806afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 --2- 172.21.15.16:0/2364246547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc25806afe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc25400b0d0 tx=0x7fc25400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 -- 172.21.15.16:0/2364246547 shutdown_connections 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 --2- 172.21.15.16:0/2364246547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 0x7fc25810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 --2- 172.21.15.16:0/2364246547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc25806afe0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 --2- 172.21.15.16:0/2364246547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc25806b5b0 0x7fc258106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 -- 172.21.15.16:0/2364246547 >> 172.21.15.16:0/2364246547 conn(0x7fc2580771a0 msgr2=0x7fc2580775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 -- 172.21.15.16:0/2364246547 shutdown_connections 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.809+0000 7fc2607c5700 1 -- 172.21.15.16:0/2364246547 wait complete. 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 Processor -- start 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 -- start start 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc258101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc25806b5b0 0x7fc258101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:18.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 0x7fc258101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc258112480 con 0x7fc25806ac00 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc258112300 con 0x7fc2581075e0 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc2607c5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc258112600 con 0x7fc25806b5b0 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc258101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc258101230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43424/0 (socket says 172.21.15.16:43424) 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 -- 172.21.15.16:0/1347790389 learned_addr learned my addr 172.21.15.16:0/1347790389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25dd60700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc25806b5b0 0x7fc258101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25ed62700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 0x7fc258101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 -- 172.21.15.16:0/1347790389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc25806b5b0 msgr2=0x7fc258101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc25806b5b0 0x7fc258101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 -- 172.21.15.16:0/1347790389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 msgr2=0x7fc258101cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 0x7fc258101cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 -- 172.21.15.16:0/1347790389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2581a6ec0 con 0x7fc25806ac00 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25ed62700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 0x7fc258101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc25e561700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc258101230 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc254000f80 tx=0x7fc254007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.810+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254021020 con 0x7fc25806ac00 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.811+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc25401db20 con 0x7fc25806ac00 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.811+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2581a7150 con 0x7fc25806ac00 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.811+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25401bb70 con 0x7fc25806ac00 2024-04-25T01:26:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.811+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2581a7650 con 0x7fc25806ac00 2024-04-25T01:26:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.811+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc25810a790 con 0x7fc25806ac00 2024-04-25T01:26:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.813+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc25401bd10 con 0x7fc25806ac00 2024-04-25T01:26:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.813+0000 7fc24f7fe700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc2440a2d30 0x7fc2440a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.813+0000 7fc25dd60700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc2440a2d30 0x7fc2440a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.813+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc25400f020 con 0x7fc25806ac00 2024-04-25T01:26:18.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.814+0000 7fc25dd60700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc2440a2d30 0x7fc2440a51f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc248006fd0 tx=0x7fc248008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:18.817+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc25409d920 con 0x7fc25806ac00 2024-04-25T01:26:19.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.027+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc258102760 con 0x7fc25806ac00 2024-04-25T01:26:19.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.027+0000 7fc24f7fe700 1 -- 172.21.15.16:0/1347790389 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc25409d220 con 0x7fc25806ac00 2024-04-25T01:26:19.007 INFO:teuthology.orchestra.run.smithi016.stdout:197568495669 2024-04-25T01:26:19.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc2440a2d30 msgr2=0x7fc2440a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc2440a2d30 0x7fc2440a51f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc248006fd0 tx=0x7fc248008040 comp rx=0 tx=0).stop 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 msgr2=0x7fc258101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc258101230 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc254000f80 tx=0x7fc254007ad0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 shutdown_connections 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc2440a2d30 0x7fc2440a51f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2581075e0 0x7fc258101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25806ac00 0x7fc258101230 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 --2- 172.21.15.16:0/1347790389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc25806b5b0 0x7fc258101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 >> 172.21.15.16:0/1347790389 conn(0x7fc2580771a0 msgr2=0x7fc2580ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 shutdown_connections 2024-04-25T01:26:19.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.030+0000 7fc2607c5700 1 -- 172.21.15.16:0/1347790389 wait complete. 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 -- 172.21.15.16:0/799897114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 msgr2=0x7fd0341015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 --2- 172.21.15.16:0/799897114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd0341015a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd028009990 tx=0x7fd028009ca0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 -- 172.21.15.16:0/799897114 shutdown_connections 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 --2- 172.21.15.16:0/799897114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd034101ae0 0x7fd03410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 --2- 172.21.15.16:0/799897114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034107190 0x7fd034107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 --2- 172.21.15.16:0/799897114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd0341015a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 -- 172.21.15.16:0/799897114 >> 172.21.15.16:0/799897114 conn(0x7fd0340780a0 msgr2=0x7fd0340784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 -- 172.21.15.16:0/799897114 shutdown_connections 2024-04-25T01:26:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.055+0000 7fd039913700 1 -- 172.21.15.16:0/799897114 wait complete. 2024-04-25T01:26:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 Processor -- start 2024-04-25T01:26:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 -- start start 2024-04-25T01:26:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd034102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034101ae0 0x7fd034103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd034107190 0x7fd034103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:19.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd034111300 con 0x7fd034101120 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd034111180 con 0x7fd034107190 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd039913700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd034111480 con 0x7fd034101ae0 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd032ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd034102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd032ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd034102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43448/0 (socket says 172.21.15.16:43448) 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.056+0000 7fd032ffd700 1 -- 172.21.15.16:0/3846662706 learned_addr learned my addr 172.21.15.16:0/3846662706 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd0327fc700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034101ae0 0x7fd034103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd0337fe700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd034107190 0x7fd034103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd032ffd700 1 -- 172.21.15.16:0/3846662706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034101ae0 msgr2=0x7fd034103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd032ffd700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034101ae0 0x7fd034103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd032ffd700 1 -- 172.21.15.16:0/3846662706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd034107190 msgr2=0x7fd034103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd032ffd700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd034107190 0x7fd034103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd032ffd700 1 -- 172.21.15.16:0/3846662706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0341a6e50 con 0x7fd034101120 2024-04-25T01:26:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd032ffd700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd034102e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd02400d950 tx=0x7fd02400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024016070 con 0x7fd034101120 2024-04-25T01:26:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd024010460 con 0x7fd034101120 2024-04-25T01:26:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.057+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0341a7140 con 0x7fd034101120 2024-04-25T01:26:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.058+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02400f620 con 0x7fd034101120 2024-04-25T01:26:19.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.058+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0341a7680 con 0x7fd034101120 2024-04-25T01:26:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.059+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd034068f20 con 0x7fd034101120 2024-04-25T01:26:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.060+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd024019070 con 0x7fd034101120 2024-04-25T01:26:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.061+0000 7fd038911700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd01c0a2cb0 0x7fd01c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.061+0000 7fd0327fc700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd01c0a2cb0 0x7fd01c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.061+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd0240cf1e0 con 0x7fd034101120 2024-04-25T01:26:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.061+0000 7fd0327fc700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd01c0a2cb0 0x7fd01c0a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd02800b2e0 tx=0x7fd02801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.065+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd02409bf60 con 0x7fd034101120 2024-04-25T01:26:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.276+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fd0341a7930 con 0x7fd034101120 2024-04-25T01:26:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.276+0000 7fd038911700 1 -- 172.21.15.16:0/3846662706 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd02409b860 con 0x7fd034101120 2024-04-25T01:26:19.257 INFO:teuthology.orchestra.run.smithi016.stdout:287762808869 2024-04-25T01:26:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd01c0a2cb0 msgr2=0x7fd01c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd01c0a2cb0 0x7fd01c0a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd02800b2e0 tx=0x7fd02801a040 comp rx=0 tx=0).stop 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 msgr2=0x7fd034102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd034102e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd02400d950 tx=0x7fd02400dd10 comp rx=0 tx=0).stop 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 shutdown_connections 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd01c0a2cb0 0x7fd01c0a5170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd034107190 0x7fd034103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd034101120 0x7fd034102e00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 --2- 172.21.15.16:0/3846662706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034101ae0 0x7fd034103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 >> 172.21.15.16:0/3846662706 conn(0x7fd0340780a0 msgr2=0x7fd034100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 shutdown_connections 2024-04-25T01:26:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:19.279+0000 7fd039913700 1 -- 172.21.15.16:0/3846662706 wait complete. 2024-04-25T01:26:19.465 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:19 smithi038 ceph-mon[32841]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:19.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:19 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1347790389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:26:19.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:19 smithi156 ceph-mon[32631]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:19.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:19 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1347790389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:26:19.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:19 smithi016 ceph-mon[29153]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:19.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:19 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1347790389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:26:20.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:20 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3846662706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:26:20.465 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:20 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3846662706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:26:20.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:20 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3846662706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:26:21.230 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808869 for osd.9 2024-04-25T01:26:21.230 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:21.453 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:21 smithi016 ceph-mon[29153]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:21.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:21 smithi038 ceph-mon[32841]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:21.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:21 smithi156 ceph-mon[32631]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:23.408 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:23.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:23 smithi156 ceph-mon[32631]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:23.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:23 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/614459411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:26:23.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:23 smithi016 ceph-mon[29153]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:23.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:23 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/614459411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:26:23.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:23 smithi038 ceph-mon[32841]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:23.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:23 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/614459411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:26:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 -- 172.21.15.16:0/2752490708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75041075d0 msgr2=0x7f750410f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 --2- 172.21.15.16:0/2752490708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75041075d0 0x7f750410f1a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00b0a0 tx=0x7f74fc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 -- 172.21.15.16:0/2752490708 shutdown_connections 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 --2- 172.21.15.16:0/2752490708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75041075d0 0x7f750410f1a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 --2- 172.21.15.16:0/2752490708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7504102510 0x7f75041028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 --2- 172.21.15.16:0/2752490708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7504102ec0 0x7f7504106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.749+0000 7f75092ed700 1 -- 172.21.15.16:0/2752490708 >> 172.21.15.16:0/2752490708 conn(0x7f75040793e0 msgr2=0x7f75040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.750+0000 7f75092ed700 1 -- 172.21.15.16:0/2752490708 shutdown_connections 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.750+0000 7f75092ed700 1 -- 172.21.15.16:0/2752490708 wait complete. 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 Processor -- start 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 -- start start 2024-04-25T01:26:24.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 0x7f750419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7504102ec0 0x7f750419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75041075d0 0x7f75041986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7504112460 con 0x7f7504102510 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75041122e0 con 0x7f75041075d0 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.751+0000 7f75092ed700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f75041125e0 con 0x7f7504102ec0 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 0x7f750419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 0x7f750419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43532/0 (socket says 172.21.15.16:43532) 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 -- 172.21.15.16:0/614459411 learned_addr learned my addr 172.21.15.16:0/614459411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f75037fe700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75041075d0 0x7f75041986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:24.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f75027fc700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7504102ec0 0x7f750419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:24.047 INFO:teuthology.orchestra.run.smithi016.stdout:55834574927 2024-04-25T01:26:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 -- 172.21.15.16:0/614459411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7504102ec0 msgr2=0x7f750419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7504102ec0 0x7f750419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 -- 172.21.15.16:0/614459411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75041075d0 msgr2=0x7f75041986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75041075d0 0x7f75041986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 -- 172.21.15.16:0/614459411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7504198fa0 con 0x7f7504102510 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f7502ffd700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 0x7f750419e5f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9c0 tx=0x7f74ec00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec00cbd0 con 0x7f7504102510 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74ec00cd70 con 0x7f7504102510 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7504199290 con 0x7f7504102510 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.752+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec0056c0 con 0x7f7504102510 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.753+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a5570 con 0x7f7504102510 2024-04-25T01:26:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.753+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75041039d0 con 0x7f7504102510 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.755+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f74ec004750 con 0x7f7504102510 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.756+0000 7f74fbfff700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74f00a2cb0 0x7f74f00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.756+0000 7f75027fc700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74f00a2cb0 0x7f74f00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.756+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f74ec0cf6a0 con 0x7f7504102510 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.756+0000 7f75027fc700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74f00a2cb0 0x7f74f00a5170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7504199e80 tx=0x7f74f400d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.760+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f74ec053740 con 0x7f7504102510 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.972+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7504199460 con 0x7f7504102510 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.972+0000 7f74fbfff700 1 -- 172.21.15.16:0/614459411 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f74ec09c480 con 0x7f7504102510 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74f00a2cb0 msgr2=0x7f74f00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74f00a2cb0 0x7f74f00a5170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7504199e80 tx=0x7f74f400d4b0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 msgr2=0x7f750419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 0x7f750419e5f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9c0 tx=0x7f74ec00ecd0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 shutdown_connections 2024-04-25T01:26:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f74f00a2cb0 0x7f74f00a5170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75041075d0 0x7f75041986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7504102510 0x7f750419e5f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 --2- 172.21.15.16:0/614459411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7504102ec0 0x7f750419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 >> 172.21.15.16:0/614459411 conn(0x7f75040793e0 msgr2=0x7f7504100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 shutdown_connections 2024-04-25T01:26:24.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:22.974+0000 7f75092ed700 1 -- 172.21.15.16:0/614459411 wait complete. 2024-04-25T01:26:24.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574927 for osd.1 2024-04-25T01:26:24.698 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:25.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:25 smithi016 ceph-mon[29153]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:25.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:25 smithi156 ceph-mon[32631]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:25.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:25 smithi038 ceph-mon[32841]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.548+0000 7fe1f305a700 1 -- 172.21.15.16:0/3085681316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec1011c0 msgr2=0x7fe1ec10f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.548+0000 7fe1f305a700 1 --2- 172.21.15.16:0/3085681316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec1011c0 0x7fe1ec10f100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400b0a0 tx=0x7fe1e400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.548+0000 7fe1f305a700 1 -- 172.21.15.16:0/3085681316 shutdown_connections 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.548+0000 7fe1f305a700 1 --2- 172.21.15.16:0/3085681316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec1011c0 0x7fe1ec10f100 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.548+0000 7fe1f305a700 1 --2- 172.21.15.16:0/3085681316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec1082c0 0x7fe1ec1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 --2- 172.21.15.16:0/3085681316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1ec100820 0x7fe1ec100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 -- 172.21.15.16:0/3085681316 >> 172.21.15.16:0/3085681316 conn(0x7fe1ec0777a0 msgr2=0x7fe1ec0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 -- 172.21.15.16:0/3085681316 shutdown_connections 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 -- 172.21.15.16:0/3085681316 wait complete. 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 Processor -- start 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 -- start start 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 0x7fe1ec101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.549+0000 7fe1f305a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec1011c0 0x7fe1ec101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f305a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1ec1082c0 0x7fe1ec102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f305a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec112470 con 0x7fe1ec100820 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f305a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1122f0 con 0x7fe1ec1011c0 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f305a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1125f0 con 0x7fe1ec1082c0 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 0x7fe1ec101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 0x7fe1ec101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43552/0 (socket says 172.21.15.16:43552) 2024-04-25T01:26:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 -- 172.21.15.16:0/2982639245 learned_addr learned my addr 172.21.15.16:0/2982639245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1ebfff700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec1011c0 0x7fe1ec101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 -- 172.21.15.16:0/2982639245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1ec1082c0 msgr2=0x7fe1ec102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f15f7700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1ec1082c0 0x7fe1ec102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1ec1082c0 0x7fe1ec102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 -- 172.21.15.16:0/2982639245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec1011c0 msgr2=0x7fe1ec101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec1011c0 0x7fe1ec101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 -- 172.21.15.16:0/2982639245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ec1a6e20 con 0x7fe1ec100820 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f0df6700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 0x7fe1ec101a00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000d950 tx=0x7fe1e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0016070 con 0x7fe1ec100820 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ec1a7110 con 0x7fe1ec100820 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1e0009df0 con 0x7fe1ec100820 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.550+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000f3c0 con 0x7fe1ec100820 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.551+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ec1a7650 con 0x7fe1ec100820 2024-04-25T01:26:26.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.551+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1ec068f20 con 0x7fe1ec100820 2024-04-25T01:26:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.553+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe1e0019070 con 0x7fe1ec100820 2024-04-25T01:26:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.553+0000 7fe1e9ffb700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe1dc0a2cb0 0x7fe1dc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.554+0000 7fe1ebfff700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe1dc0a2cb0 0x7fe1dc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.554+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe1e00cee00 con 0x7fe1ec100820 2024-04-25T01:26:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.554+0000 7fe1ebfff700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe1dc0a2cb0 0x7fe1dc0a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec107a30 tx=0x7fe1d8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.558+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe1e009bbe0 con 0x7fe1ec100820 2024-04-25T01:26:26.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.771+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe1ec02db30 con 0x7fe1ec100820 2024-04-25T01:26:26.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.771+0000 7fe1e9ffb700 1 -- 172.21.15.16:0/2982639245 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe1e009b4e0 con 0x7fe1ec100820 2024-04-25T01:26:26.751 INFO:teuthology.orchestra.run.smithi016.stdout:77309411403 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe1dc0a2cb0 msgr2=0x7fe1dc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe1dc0a2cb0 0x7fe1dc0a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec107a30 tx=0x7fe1d8009190 comp rx=0 tx=0).stop 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 msgr2=0x7fe1ec101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 0x7fe1ec101a00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000d950 tx=0x7fe1e000dc60 comp rx=0 tx=0).stop 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 shutdown_connections 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe1dc0a2cb0 0x7fe1dc0a5170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1ec1082c0 0x7fe1ec102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1ec100820 0x7fe1ec101a00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 --2- 172.21.15.16:0/2982639245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1ec1011c0 0x7fe1ec101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:26.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.773+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 >> 172.21.15.16:0/2982639245 conn(0x7fe1ec0777a0 msgr2=0x7fe1ec0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:26.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.774+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 shutdown_connections 2024-04-25T01:26:26.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.774+0000 7fe1f305a700 1 -- 172.21.15.16:0/2982639245 wait complete. 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 -- 172.21.15.16:0/2753740167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584406b170 msgr2=0x7f584410f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 --2- 172.21.15.16:0/2753740167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584406b170 0x7f584410f0e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f583c00b0a0 tx=0x7f583c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 -- 172.21.15.16:0/2753740167 shutdown_connections 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 --2- 172.21.15.16:0/2753740167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584406b170 0x7f584410f0e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 --2- 172.21.15.16:0/2753740167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58441082a0 0x7f5844108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 --2- 172.21.15.16:0/2753740167 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f584406a7d0 0x7f584406ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 -- 172.21.15.16:0/2753740167 >> 172.21.15.16:0/2753740167 conn(0x7f58440780a0 msgr2=0x7f58440784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 -- 172.21.15.16:0/2753740167 shutdown_connections 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.793+0000 7f584b3ab700 1 -- 172.21.15.16:0/2753740167 wait complete. 2024-04-25T01:26:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 Processor -- start 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 -- start start 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f584406a7d0 0x7f58441011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f584406b170 0x7f5844101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 0x7f5844101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f58441123d0 con 0x7f58441082a0 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5844112250 con 0x7f584406a7d0 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f584b3ab700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5844112550 con 0x7f584406b170 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f5849948700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 0x7f5844101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f5849948700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 0x7f5844101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43564/0 (socket says 172.21.15.16:43564) 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f5849948700 1 -- 172.21.15.16:0/3899360152 learned_addr learned my addr 172.21.15.16:0/3899360152 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.794+0000 7f5849147700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f584406a7d0 0x7f58441011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5848946700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f584406b170 0x7f5844101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5849948700 1 -- 172.21.15.16:0/3899360152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f584406b170 msgr2=0x7f5844101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:27.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5849948700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f584406b170 0x7f5844101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5849948700 1 -- 172.21.15.16:0/3899360152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f584406a7d0 msgr2=0x7f58441011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5849948700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f584406a7d0 0x7f58441011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5849948700 1 -- 172.21.15.16:0/3899360152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58441a6e90 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f5849948700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 0x7f5844101c70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f583c0062a0 tx=0x7f583c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c021070 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58441a7120 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f583c007e30 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c0046d0 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.795+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58441a7620 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.796+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5844068f20 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.797+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f583c007980 con 0x7f58441082a0 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.798+0000 7f583a7fc700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f58300a2cb0 0x7f58300a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.798+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f583c00f020 con 0x7f58441082a0 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.798+0000 7f5849147700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f58300a2cb0 0x7f58300a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.799+0000 7f5849147700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f58300a2cb0 0x7f58300a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5840009be0 tx=0x7f5840009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:26.803+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f583c02a020 con 0x7f58441082a0 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.012+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f584402db30 con 0x7f58441082a0 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.013+0000 7f583a7fc700 1 -- 172.21.15.16:0/3899360152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f583c09d480 con 0x7f58441082a0 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stdout:219043332144 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f58300a2cb0 msgr2=0x7f58300a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f58300a2cb0 0x7f58300a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5840009be0 tx=0x7f5840009260 comp rx=0 tx=0).stop 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 msgr2=0x7f5844101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 0x7f5844101c70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f583c0062a0 tx=0x7f583c004c80 comp rx=0 tx=0).stop 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 shutdown_connections 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f58300a2cb0 0x7f58300a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58441082a0 0x7f5844101c70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f584406a7d0 0x7f58441011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 --2- 172.21.15.16:0/3899360152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f584406b170 0x7f5844101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 >> 172.21.15.16:0/3899360152 conn(0x7f58440780a0 msgr2=0x7f58440ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 shutdown_connections 2024-04-25T01:26:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:27.015+0000 7f584b3ab700 1 -- 172.21.15.16:0/3899360152 wait complete. 2024-04-25T01:26:27.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:27 smithi016 ceph-mon[29153]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T01:26:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:27 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2982639245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:26:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:27 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3899360152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:26:27.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:27 smithi156 ceph-mon[32631]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T01:26:27.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:27 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2982639245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:26:27.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:27 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3899360152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:26:27.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:27 smithi038 ceph-mon[32841]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T01:26:27.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:27 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2982639245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:26:27.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:27 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3899360152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:26:28.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495669 for osd.6 2024-04-25T01:26:28.148 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:28.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 -- 172.21.15.16:0/3083593089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 msgr2=0x7fa21c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 --2- 172.21.15.16:0/3083593089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c106eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009990 tx=0x7fa20c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 -- 172.21.15.16:0/3083593089 shutdown_connections 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 --2- 172.21.15.16:0/3083593089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa21c107590 0x7fa21c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 --2- 172.21.15.16:0/3083593089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 0x7fa21c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 --2- 172.21.15.16:0/3083593089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c106eb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 -- 172.21.15.16:0/3083593089 >> 172.21.15.16:0/3083593089 conn(0x7fa21c0793a0 msgr2=0x7fa21c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 -- 172.21.15.16:0/3083593089 shutdown_connections 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.396+0000 7fa221cb4700 1 -- 172.21.15.16:0/3083593089 wait complete. 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 Processor -- start 2024-04-25T01:26:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 -- start start 2024-04-25T01:26:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 0x7fa21c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa21c107590 0x7fa21c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c1124e0 con 0x7fa21c102ec0 2024-04-25T01:26:28.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c112360 con 0x7fa21c102510 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa221cb4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c112660 con 0x7fa21c107590 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.397+0000 7fa21affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43576/0 (socket says 172.21.15.16:43576) 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 -- 172.21.15.16:0/425508718 learned_addr learned my addr 172.21.15.16:0/425508718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21b7fe700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 0x7fa21c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21bfff700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa21c107590 0x7fa21c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 -- 172.21.15.16:0/425508718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa21c107590 msgr2=0x7fa21c074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa21c107590 0x7fa21c074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 -- 172.21.15.16:0/425508718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 msgr2=0x7fa21c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 0x7fa21c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 -- 172.21.15.16:0/425508718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21c1a6f40 con 0x7fa21c102ec0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21b7fe700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 0x7fa21c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa21affd700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c074a70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa20c000c00 tx=0x7fa20c003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c027070 con 0x7fa21c102ec0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa20c00bdc0 con 0x7fa21c102ec0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21c1a71d0 con 0x7fa21c102ec0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20c00f460 con 0x7fa21c102ec0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.398+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21c1a7680 con 0x7fa21c102ec0 2024-04-25T01:26:28.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.399+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa21c103990 con 0x7fa21c102ec0 2024-04-25T01:26:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.400+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa20c02a070 con 0x7fa21c102ec0 2024-04-25T01:26:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.401+0000 7fa218ff9700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2040a2d30 0x7fa2040a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.401+0000 7fa21b7fe700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2040a2d30 0x7fa2040a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.401+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa20c0d0040 con 0x7fa21c102ec0 2024-04-25T01:26:28.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.402+0000 7fa21b7fe700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2040a2d30 0x7fa2040a51f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa2100099d0 tx=0x7fa210008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:28.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.406+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa20c013070 con 0x7fa21c102ec0 2024-04-25T01:26:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 -- 172.21.15.16:0/4143969449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 msgr2=0x7f91e4102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 --2- 172.21.15.16:0/4143969449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4102760 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0099c0 tx=0x7f91cc009cd0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 -- 172.21.15.16:0/4143969449 shutdown_connections 2024-04-25T01:26:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 --2- 172.21.15.16:0/4143969449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e4102ca0 0x7f91e410f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 --2- 172.21.15.16:0/4143969449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91e41082f0 0x7f91e41086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 --2- 172.21.15.16:0/4143969449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4102760 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 -- 172.21.15.16:0/4143969449 >> 172.21.15.16:0/4143969449 conn(0x7f91e4079390 msgr2=0x7f91e40797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 -- 172.21.15.16:0/4143969449 shutdown_connections 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.598+0000 7f91ea150700 1 -- 172.21.15.16:0/4143969449 wait complete. 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 Processor -- start 2024-04-25T01:26:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 -- start start 2024-04-25T01:26:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91e4102ca0 0x7f91e4074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e41082f0 0x7f91e4074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4112510 con 0x7f91e41022e0 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4112390 con 0x7f91e41082f0 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.599+0000 7f91ea150700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4112690 con 0x7f91e4102ca0 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43596/0 (socket says 172.21.15.16:43596) 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 -- 172.21.15.16:0/3030536119 learned_addr learned my addr 172.21.15.16:0/3030536119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e3fff700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e41082f0 0x7f91e4074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91daffd700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91e4102ca0 0x7f91e4074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 -- 172.21.15.16:0/3030536119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91e4102ca0 msgr2=0x7f91e4074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91e4102ca0 0x7f91e4074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 -- 172.21.15.16:0/3030536119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e41082f0 msgr2=0x7f91e4074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e41082f0 0x7f91e4074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 -- 172.21.15.16:0/3030536119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e41a6f10 con 0x7f91e41022e0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e37fe700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4074530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c970 tx=0x7f91d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d40078d0 con 0x7f91e41022e0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91d4018480 con 0x7f91e41022e0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d400f6b0 con 0x7f91e41022e0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.600+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e41a7200 con 0x7f91e41022e0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.601+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e41a76c0 con 0x7f91e41022e0 2024-04-25T01:26:28.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.601+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e4068f20 con 0x7f91e41022e0 2024-04-25T01:26:28.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.603+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f91d4018630 con 0x7f91e41022e0 2024-04-25T01:26:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.604+0000 7f91e17fa700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d00a2d30 0x7f91d00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.604+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f91d40cf630 con 0x7f91e41022e0 2024-04-25T01:26:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.604+0000 7f91daffd700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d00a2d30 0x7f91d00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.605+0000 7f91daffd700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d00a2d30 0x7f91d00a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f91e4075fc0 tx=0x7f91cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:28.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.611+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f91d409c4d0 con 0x7f91e41022e0 2024-04-25T01:26:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.622+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa21c02db30 con 0x7fa21c102ec0 2024-04-25T01:26:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.622+0000 7fa218ff9700 1 -- 172.21.15.16:0/425508718 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa20c09ceb0 con 0x7fa21c102ec0 2024-04-25T01:26:28.602 INFO:teuthology.orchestra.run.smithi016.stdout:343597383709 2024-04-25T01:26:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.624+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2040a2d30 msgr2=0x7fa2040a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.624+0000 7fa221cb4700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2040a2d30 0x7fa2040a51f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa2100099d0 tx=0x7fa210008040 comp rx=0 tx=0).stop 2024-04-25T01:26:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.624+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 msgr2=0x7fa21c074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.624+0000 7fa221cb4700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c074a70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa20c000c00 tx=0x7fa20c003b30 comp rx=0 tx=0).stop 2024-04-25T01:26:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 shutdown_connections 2024-04-25T01:26:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2040a2d30 0x7fa2040a51f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa21c107590 0x7fa21c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa21c102510 0x7fa21c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 --2- 172.21.15.16:0/425508718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21c102ec0 0x7fa21c074a70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 >> 172.21.15.16:0/425508718 conn(0x7fa21c0793a0 msgr2=0x7fa21c101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 shutdown_connections 2024-04-25T01:26:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.625+0000 7fa221cb4700 1 -- 172.21.15.16:0/425508718 wait complete. 2024-04-25T01:26:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.830+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f91e4075a60 con 0x7f91e41022e0 2024-04-25T01:26:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.830+0000 7f91e17fa700 1 -- 172.21.15.16:0/3030536119 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f91d409bdd0 con 0x7f91e41022e0 2024-04-25T01:26:28.810 INFO:teuthology.orchestra.run.smithi016.stdout:163208757307 2024-04-25T01:26:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.832+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d00a2d30 msgr2=0x7f91d00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.832+0000 7f91ea150700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d00a2d30 0x7f91d00a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f91e4075fc0 tx=0x7f91cc005fb0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.832+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 msgr2=0x7f91e4074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:28.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.832+0000 7f91ea150700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4074530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c970 tx=0x7f91d400cc80 comp rx=0 tx=0).stop 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 shutdown_connections 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d00a2d30 0x7f91d00a51f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e41082f0 0x7f91e4074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e41022e0 0x7f91e4074530 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 --2- 172.21.15.16:0/3030536119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91e4102ca0 0x7f91e4074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 >> 172.21.15.16:0/3030536119 conn(0x7f91e4079390 msgr2=0x7f91e4101010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 shutdown_connections 2024-04-25T01:26:28.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:28.833+0000 7f91ea150700 1 -- 172.21.15.16:0/3030536119 wait complete. 2024-04-25T01:26:29.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:29 smithi156 ceph-mon[32631]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T01:26:29.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/425508718' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:26:29.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3030536119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:26:29.519 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:29 smithi016 ceph-mon[29153]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T01:26:29.519 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/425508718' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:26:29.520 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3030536119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:26:29.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:29 smithi038 ceph-mon[32841]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T01:26:29.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/425508718' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:26:29.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3030536119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:26:29.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332144 for osd.7 2024-04-25T01:26:29.940 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:30.495 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411397 got 77309411403 for osd.2 2024-04-25T01:26:30.496 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:31.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:31 smithi156 ceph-mon[32631]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T01:26:31.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:31 smithi156 ceph-mon[32631]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T01:26:31.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:31 smithi016 ceph-mon[29153]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T01:26:31.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:31 smithi016 ceph-mon[29153]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T01:26:31.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:31 smithi038 ceph-mon[32841]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T01:26:31.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:31 smithi038 ceph-mon[32841]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.721+0000 7f2d5e9df700 1 -- 172.21.15.16:0/2021917471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 msgr2=0x7f2d5810f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.721+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/2021917471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d5810f170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000b0a0 tx=0x7f2d5000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 -- 172.21.15.16:0/2021917471 shutdown_connections 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/2021917471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d5810f170 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/2021917471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d581024e0 0x7f2d581028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/2021917471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d58102e90 0x7f2d58106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 -- 172.21.15.16:0/2021917471 >> 172.21.15.16:0/2021917471 conn(0x7f2d580793e0 msgr2=0x7f2d580797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 -- 172.21.15.16:0/2021917471 shutdown_connections 2024-04-25T01:26:32.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 -- 172.21.15.16:0/2021917471 wait complete. 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 Processor -- start 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.722+0000 7f2d5e9df700 1 -- start start 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5e9df700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d581024e0 0x7f2d58074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5e9df700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d58102e90 0x7f2d58074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5e9df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d58074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5e9df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2d58112430 con 0x7f2d581075a0 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5e9df700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d581122b0 con 0x7f2d58102e90 2024-04-25T01:26:32.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5e9df700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d581125b0 con 0x7f2d581024e0 2024-04-25T01:26:32.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d58074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:32.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d58074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54402/0 (socket says 172.21.15.16:54402) 2024-04-25T01:26:32.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 -- 172.21.15.16:0/4235691175 learned_addr learned my addr 172.21.15.16:0/4235691175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:32.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d577fe700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d58102e90 0x7f2d58074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:32.035 INFO:teuthology.orchestra.run.smithi016.stdout:253403070508 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d57fff700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d581024e0 0x7f2d58074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 -- 172.21.15.16:0/4235691175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d581024e0 msgr2=0x7f2d58074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d581024e0 0x7f2d58074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 -- 172.21.15.16:0/4235691175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d58102e90 msgr2=0x7f2d58074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d58102e90 0x7f2d58074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.723+0000 7f2d5cf7c700 1 -- 172.21.15.16:0/4235691175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d581a6e30 con 0x7f2d581075a0 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.724+0000 7f2d5cf7c700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d58074fb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000b830 tx=0x7f2d50004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.724+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50021070 con 0x7f2d581075a0 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.724+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d50007e30 con 0x7f2d581075a0 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.724+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50004730 con 0x7f2d581075a0 2024-04-25T01:26:32.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.724+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d581a70c0 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.724+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d581a7520 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.725+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d58068f20 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.726+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2d50003e50 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.727+0000 7f2d557fa700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2d400a2c20 0x7f2d400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.727+0000 7f2d57fff700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2d400a2c20 0x7f2d400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.727+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2d5000f020 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.728+0000 7f2d57fff700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2d400a2c20 0x7f2d400a50e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d48005fd0 tx=0x7f2d4800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.732+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2d5002a020 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.940+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f2d5802dcf0 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.941+0000 7f2d557fa700 1 -- 172.21.15.16:0/4235691175 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2d5009d500 con 0x7f2d581075a0 2024-04-25T01:26:32.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2d400a2c20 msgr2=0x7f2d400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2d400a2c20 0x7f2d400a50e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d48005fd0 tx=0x7f2d4800a560 comp rx=0 tx=0).stop 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 msgr2=0x7f2d58074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d58074fb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000b830 tx=0x7f2d50004ce0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 shutdown_connections 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f2d400a2c20 0x7f2d400a50e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d581075a0 0x7f2d58074fb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d581024e0 0x7f2d58074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 --2- 172.21.15.16:0/4235691175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d58102e90 0x7f2d58074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:32.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 >> 172.21.15.16:0/4235691175 conn(0x7f2d580793e0 msgr2=0x7f2d58101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:32.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 shutdown_connections 2024-04-25T01:26:32.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:31.943+0000 7f2d5e9df700 1 -- 172.21.15.16:0/4235691175 wait complete. 2024-04-25T01:26:32.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:32 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/4235691175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:26:32.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:32 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4235691175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:26:32.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:32 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/4235691175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:26:32.733 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070508 for osd.8 2024-04-25T01:26:32.734 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:33.206 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757307 for osd.5 2024-04-25T01:26:33.207 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:33.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:33 smithi016 ceph-mon[29153]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:33.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:33 smithi156 ceph-mon[32631]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:33.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:33 smithi038 ceph-mon[32841]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 -- 172.21.15.16:0/597864774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc406a8e0 msgr2=0x7f6cc410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/597864774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc406a8e0 0x7f6cc410f0f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00b0a0 tx=0x7f6cbc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 -- 172.21.15.16:0/597864774 shutdown_connections 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/597864774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc406a8e0 0x7f6cc410f0f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/597864774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cc41082b0 0x7f6cc4108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/597864774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cc4069f40 0x7f6cc406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 -- 172.21.15.16:0/597864774 >> 172.21.15.16:0/597864774 conn(0x7f6cc40780e0 msgr2=0x7f6cc40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:33.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 -- 172.21.15.16:0/597864774 shutdown_connections 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.924+0000 7f6cc8af7700 1 -- 172.21.15.16:0/597864774 wait complete. 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 Processor -- start 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 -- start start 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cc4069f40 0x7f6cc419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cc406a8e0 0x7f6cc419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 0x7f6cc4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41123e0 con 0x7f6cc41082b0 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc4112260 con 0x7f6cc406a8e0 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc8af7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc4112560 con 0x7f6cc4069f40 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 0x7f6cc4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cc4069f40 0x7f6cc419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 0x7f6cc4198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54418/0 (socket says 172.21.15.16:54418) 2024-04-25T01:26:33.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.925+0000 7f6cc2d9d700 1 -- 172.21.15.16:0/3454992424 learned_addr learned my addr 172.21.15.16:0/3454992424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc1d9b700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cc406a8e0 0x7f6cc419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc2d9d700 1 -- 172.21.15.16:0/3454992424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cc4069f40 msgr2=0x7f6cc419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc2d9d700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cc4069f40 0x7f6cc419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc2d9d700 1 -- 172.21.15.16:0/3454992424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cc406a8e0 msgr2=0x7f6cc419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc2d9d700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cc406a8e0 0x7f6cc419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc2d9d700 1 -- 172.21.15.16:0/3454992424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc4199020 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc2d9d700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 0x7f6cc4198740 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00b830 tx=0x7f6cbc004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc021070 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc41992b0 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cbc007e30 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0047d0 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.926+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc4101300 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.927+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc4199620 con 0x7f6cc41082b0 2024-04-25T01:26:33.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.928+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6cbc007980 con 0x7f6cc41082b0 2024-04-25T01:26:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.929+0000 7f6cb37fe700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6cac0a2c20 0x7f6cac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.929+0000 7f6cc259c700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6cac0a2c20 0x7f6cac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.929+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6cbc00f020 con 0x7f6cc41082b0 2024-04-25T01:26:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.929+0000 7f6cc259c700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6cac0a2c20 0x7f6cac0a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4004150 tx=0x7f6cb400f7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:33.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:33.934+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6cc4199620 con 0x7f6cc41082b0 2024-04-25T01:26:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.159+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6cc4068f20 con 0x7f6cc41082b0 2024-04-25T01:26:34.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.159+0000 7f6cb37fe700 1 -- 172.21.15.16:0/3454992424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6cbc02a020 con 0x7f6cc41082b0 2024-04-25T01:26:34.140 INFO:teuthology.orchestra.run.smithi016.stdout:34359738454 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.161+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6cac0a2c20 msgr2=0x7f6cac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.161+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6cac0a2c20 0x7f6cac0a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4004150 tx=0x7f6cb400f7d0 comp rx=0 tx=0).stop 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.161+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 msgr2=0x7f6cc4198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 0x7f6cc4198740 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00b830 tx=0x7f6cbc004ce0 comp rx=0 tx=0).stop 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 shutdown_connections 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6cac0a2c20 0x7f6cac0a50e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc41082b0 0x7f6cc4198740 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cc4069f40 0x7f6cc419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:34.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 --2- 172.21.15.16:0/3454992424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6cc406a8e0 0x7f6cc419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:34.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 >> 172.21.15.16:0/3454992424 conn(0x7f6cc40780e0 msgr2=0x7f6cc40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:34.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 shutdown_connections 2024-04-25T01:26:34.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:34.162+0000 7f6cc8af7700 1 -- 172.21.15.16:0/3454992424 wait complete. 2024-04-25T01:26:34.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:34 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3454992424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:26:34.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:34 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3454992424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:26:34.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:34 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3454992424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:26:34.950 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738454 for osd.0 2024-04-25T01:26:34.950 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:35.285 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383704 got 343597383709 for osd.11 2024-04-25T01:26:35.286 DEBUG:teuthology.parallel:result is None 2024-04-25T01:26:35.286 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T01:26:35.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:26:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:35.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:35 smithi016 ceph-mon[29153]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:26:35.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:35 smithi038 ceph-mon[32841]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:26:35.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:35 smithi156 ceph-mon[32631]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:26:37.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 -- 172.21.15.16:0/3557899059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517406b370 msgr2=0x7f517410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 --2- 172.21.15.16:0/3557899059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517406b370 0x7f517410f0f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f516c00b0a0 tx=0x7f516c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 -- 172.21.15.16:0/3557899059 shutdown_connections 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 --2- 172.21.15.16:0/3557899059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f517406b370 0x7f517410f0f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 --2- 172.21.15.16:0/3557899059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51741082b0 0x7f5174108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 --2- 172.21.15.16:0/3557899059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f517406a9d0 0x7f517406ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 -- 172.21.15.16:0/3557899059 >> 172.21.15.16:0/3557899059 conn(0x7f51740780e0 msgr2=0x7f51740784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 -- 172.21.15.16:0/3557899059 shutdown_connections 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.272+0000 7f5179b57700 1 -- 172.21.15.16:0/3557899059 wait complete. 2024-04-25T01:26:37.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.273+0000 7f5179b57700 1 Processor -- start 2024-04-25T01:26:37.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.273+0000 7f5179b57700 1 -- start start 2024-04-25T01:26:37.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.273+0000 7f5179b57700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f517406a9d0 0x7f517419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:37.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.273+0000 7f5179b57700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f517406b370 0x7f517419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5179b57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 0x7f5174198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5179b57700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51741123e0 con 0x7f51741082b0 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5179b57700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5174112260 con 0x7f517406b370 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5179b57700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5174112560 con 0x7f517406a9d0 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 0x7f5174198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 0x7f5174198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54438/0 (socket says 172.21.15.16:54438) 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 -- 172.21.15.16:0/2709221598 learned_addr learned my addr 172.21.15.16:0/2709221598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5172ffd700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f517406b370 0x7f517419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f51737fe700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f517406a9d0 0x7f517419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 -- 172.21.15.16:0/2709221598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f517406a9d0 msgr2=0x7f517419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f517406a9d0 0x7f517419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 -- 172.21.15.16:0/2709221598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f517406b370 msgr2=0x7f517419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f517406b370 0x7f517419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 -- 172.21.15.16:0/2709221598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5174199020 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.274+0000 7f5173fff700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 0x7f5174198740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f516c00b830 tx=0x7f516c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.275+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c021070 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.275+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51741992b0 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.275+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f516c007e30 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.275+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c004700 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.275+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51741a55f0 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.276+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5174103960 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.277+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f516c007980 con 0x7f51741082b0 2024-04-25T01:26:37.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.278+0000 7f5170ff9700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f515c0a2ca0 0x7f515c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:37.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.278+0000 7f51737fe700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f515c0a2ca0 0x7f515c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:37.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.278+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f516c00f020 con 0x7f51741082b0 2024-04-25T01:26:37.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.278+0000 7f51737fe700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f515c0a2ca0 0x7f515c0a5160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f51640045b0 tx=0x7f516400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:37.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.282+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f516c02a020 con 0x7f51741082b0 2024-04-25T01:26:37.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.495+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5174199c70 con 0x7f515c0a2ca0 2024-04-25T01:26:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.503+0000 7f5170ff9700 1 -- 172.21.15.16:0/2709221598 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259253 (secure 0 0 0) 0x7f5174199c70 con 0x7f515c0a2ca0 2024-04-25T01:26:37.487 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.505+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f515c0a2ca0 msgr2=0x7f515c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f515c0a2ca0 0x7f515c0a5160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f51640045b0 tx=0x7f516400b3c0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 msgr2=0x7f5174198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 0x7f5174198740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f516c00b830 tx=0x7f516c004ce0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 shutdown_connections 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f515c0a2ca0 0x7f515c0a5160 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51741082b0 0x7f5174198740 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f517406a9d0 0x7f517419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 --2- 172.21.15.16:0/2709221598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f517406b370 0x7f517419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:37.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 >> 172.21.15.16:0/2709221598 conn(0x7f51740780e0 msgr2=0x7f51740ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:37.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 shutdown_connections 2024-04-25T01:26:37.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:37.506+0000 7f5179b57700 1 -- 172.21.15.16:0/2709221598 wait complete. 2024-04-25T01:26:37.492 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-25T01:26:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:37 smithi016 ceph-mon[29153]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:37 smithi038 ceph-mon[32841]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:37.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:37.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:37.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:37 smithi156 ceph-mon[32631]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:37.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:37.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:37.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:37.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:26:38.214 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-04-25T01:26:35.629339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":584,"num_read_kb":406,"num_write":346,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1241894,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":780,"ondisk_log_size":780,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368988,"kb_used_data":11516,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572836,"statfs":{"total":1151940427776,"available":1151562584064,"internally_reserved":0,"allocated":11792384,"data_stored":7437777,"data_compressed":192672,"data_compressed_allocated":253952,"data_compressed_original":2039808,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":41,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003111"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643363+0000","last_change":"2024-04-25T01:25:16.740766+0000","last_active":"2024-04-25T01:26:29.643363+0000","last_peered":"2024-04-25T01:26:29.643363+0000","last_clean":"2024-04-25T01:26:29.643363+0000","last_became_active":"2024-04-25T01:25:16.740181+0000","last_became_peered":"2024-04-25T01:25:16.740181+0000","last_unstale":"2024-04-25T01:26:29.643363+0000","last_undegraded":"2024-04-25T01:26:29.643363+0000","last_fullsized":"2024-04-25T01:26:29.643363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:59.154389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.645118+0000","last_change":"2024-04-25T01:25:16.750724+0000","last_active":"2024-04-25T01:26:29.645118+0000","last_peered":"2024-04-25T01:26:29.645118+0000","last_clean":"2024-04-25T01:26:29.645118+0000","last_became_active":"2024-04-25T01:25:16.750391+0000","last_became_peered":"2024-04-25T01:25:16.750391+0000","last_unstale":"2024-04-25T01:26:29.645118+0000","last_undegraded":"2024-04-25T01:26:29.645118+0000","last_fullsized":"2024-04-25T01:26:29.645118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:15.576020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658916+0000","last_change":"2024-04-25T01:25:16.753914+0000","last_active":"2024-04-25T01:26:29.658916+0000","last_peered":"2024-04-25T01:26:29.658916+0000","last_clean":"2024-04-25T01:26:29.658916+0000","last_became_active":"2024-04-25T01:25:16.753464+0000","last_became_peered":"2024-04-25T01:25:16.753464+0000","last_unstale":"2024-04-25T01:26:29.658916+0000","last_undegraded":"2024-04-25T01:26:29.658916+0000","last_fullsized":"2024-04-25T01:26:29.658916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:55.716948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.655867+0000","last_change":"2024-04-25T01:25:16.754586+0000","last_active":"2024-04-25T01:26:29.655867+0000","last_peered":"2024-04-25T01:26:29.655867+0000","last_clean":"2024-04-25T01:26:29.655867+0000","last_became_active":"2024-04-25T01:25:16.752260+0000","last_became_peered":"2024-04-25T01:25:16.752260+0000","last_unstale":"2024-04-25T01:26:29.655867+0000","last_undegraded":"2024-04-25T01:26:29.655867+0000","last_fullsized":"2024-04-25T01:26:29.655867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:30.935184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.629434+0000","last_change":"2024-04-25T01:25:16.741397+0000","last_active":"2024-04-25T01:26:29.629434+0000","last_peered":"2024-04-25T01:26:29.629434+0000","last_clean":"2024-04-25T01:26:29.629434+0000","last_became_active":"2024-04-25T01:25:16.741056+0000","last_became_peered":"2024-04-25T01:25:16.741056+0000","last_unstale":"2024-04-25T01:26:29.629434+0000","last_undegraded":"2024-04-25T01:26:29.629434+0000","last_fullsized":"2024-04-25T01:26:29.629434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:38.886023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.657161+0000","last_change":"2024-04-25T01:25:16.757351+0000","last_active":"2024-04-25T01:26:29.657161+0000","last_peered":"2024-04-25T01:26:29.657161+0000","last_clean":"2024-04-25T01:26:29.657161+0000","last_became_active":"2024-04-25T01:25:16.756980+0000","last_became_peered":"2024-04-25T01:25:16.756980+0000","last_unstale":"2024-04-25T01:26:29.657161+0000","last_undegraded":"2024-04-25T01:26:29.657161+0000","last_fullsized":"2024-04-25T01:26:29.657161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:18.755302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.662243+0000","last_change":"2024-04-25T01:25:16.740265+0000","last_active":"2024-04-25T01:26:29.662243+0000","last_peered":"2024-04-25T01:26:29.662243+0000","last_clean":"2024-04-25T01:26:29.662243+0000","last_became_active":"2024-04-25T01:25:16.739345+0000","last_became_peered":"2024-04-25T01:25:16.739345+0000","last_unstale":"2024-04-25T01:26:29.662243+0000","last_undegraded":"2024-04-25T01:26:29.662243+0000","last_fullsized":"2024-04-25T01:26:29.662243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:55.399650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.652116+0000","last_change":"2024-04-25T01:25:16.753159+0000","last_active":"2024-04-25T01:26:29.652116+0000","last_peered":"2024-04-25T01:26:29.652116+0000","last_clean":"2024-04-25T01:26:29.652116+0000","last_became_active":"2024-04-25T01:25:16.751863+0000","last_became_peered":"2024-04-25T01:25:16.751863+0000","last_unstale":"2024-04-25T01:26:29.652116+0000","last_undegraded":"2024-04-25T01:26:29.652116+0000","last_fullsized":"2024-04-25T01:26:29.652116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:54.218096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658976+0000","last_change":"2024-04-25T01:25:16.738727+0000","last_active":"2024-04-25T01:26:29.658976+0000","last_peered":"2024-04-25T01:26:29.658976+0000","last_clean":"2024-04-25T01:26:29.658976+0000","last_became_active":"2024-04-25T01:25:16.738390+0000","last_became_peered":"2024-04-25T01:25:16.738390+0000","last_unstale":"2024-04-25T01:26:29.658976+0000","last_undegraded":"2024-04-25T01:26:29.658976+0000","last_fullsized":"2024-04-25T01:26:29.658976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:03.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.657176+0000","last_change":"2024-04-25T01:25:16.756974+0000","last_active":"2024-04-25T01:26:29.657176+0000","last_peered":"2024-04-25T01:26:29.657176+0000","last_clean":"2024-04-25T01:26:29.657176+0000","last_became_active":"2024-04-25T01:25:16.756549+0000","last_became_peered":"2024-04-25T01:25:16.756549+0000","last_unstale":"2024-04-25T01:26:29.657176+0000","last_undegraded":"2024-04-25T01:26:29.657176+0000","last_fullsized":"2024-04-25T01:26:29.657176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:49.042132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.660437+0000","last_change":"2024-04-25T01:25:16.745728+0000","last_active":"2024-04-25T01:26:29.660437+0000","last_peered":"2024-04-25T01:26:29.660437+0000","last_clean":"2024-04-25T01:26:29.660437+0000","last_became_active":"2024-04-25T01:25:16.745091+0000","last_became_peered":"2024-04-25T01:25:16.745091+0000","last_unstale":"2024-04-25T01:26:29.660437+0000","last_undegraded":"2024-04-25T01:26:29.660437+0000","last_fullsized":"2024-04-25T01:26:29.660437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:01.622335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.651892+0000","last_change":"2024-04-25T01:25:16.755146+0000","last_active":"2024-04-25T01:26:29.651892+0000","last_peered":"2024-04-25T01:26:29.651892+0000","last_clean":"2024-04-25T01:26:29.651892+0000","last_became_active":"2024-04-25T01:25:16.752713+0000","last_became_peered":"2024-04-25T01:25:16.752713+0000","last_unstale":"2024-04-25T01:26:29.651892+0000","last_undegraded":"2024-04-25T01:26:29.651892+0000","last_fullsized":"2024-04-25T01:26:29.651892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:31.967903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.642337+0000","last_change":"2024-04-25T01:25:16.751162+0000","last_active":"2024-04-25T01:26:29.642337+0000","last_peered":"2024-04-25T01:26:29.642337+0000","last_clean":"2024-04-25T01:26:29.642337+0000","last_became_active":"2024-04-25T01:25:16.750772+0000","last_became_peered":"2024-04-25T01:25:16.750772+0000","last_unstale":"2024-04-25T01:26:29.642337+0000","last_undegraded":"2024-04-25T01:26:29.642337+0000","last_fullsized":"2024-04-25T01:26:29.642337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:39.533120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.652081+0000","last_change":"2024-04-25T01:25:16.754367+0000","last_active":"2024-04-25T01:26:29.652081+0000","last_peered":"2024-04-25T01:26:29.652081+0000","last_clean":"2024-04-25T01:26:29.652081+0000","last_became_active":"2024-04-25T01:25:16.750324+0000","last_became_peered":"2024-04-25T01:25:16.750324+0000","last_unstale":"2024-04-25T01:26:29.652081+0000","last_undegraded":"2024-04-25T01:26:29.652081+0000","last_fullsized":"2024-04-25T01:26:29.652081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:17.606187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.642673+0000","last_change":"2024-04-25T01:25:16.753111+0000","last_active":"2024-04-25T01:26:29.642673+0000","last_peered":"2024-04-25T01:26:29.642673+0000","last_clean":"2024-04-25T01:26:29.642673+0000","last_became_active":"2024-04-25T01:25:16.752749+0000","last_became_peered":"2024-04-25T01:25:16.752749+0000","last_unstale":"2024-04-25T01:26:29.642673+0000","last_undegraded":"2024-04-25T01:26:29.642673+0000","last_fullsized":"2024-04-25T01:26:29.642673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:07:28.653568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661925+0000","last_change":"2024-04-25T01:25:16.744958+0000","last_active":"2024-04-25T01:26:29.661925+0000","last_peered":"2024-04-25T01:26:29.661925+0000","last_clean":"2024-04-25T01:26:29.661925+0000","last_became_active":"2024-04-25T01:25:16.744606+0000","last_became_peered":"2024-04-25T01:25:16.744606+0000","last_unstale":"2024-04-25T01:26:29.661925+0000","last_undegraded":"2024-04-25T01:26:29.661925+0000","last_fullsized":"2024-04-25T01:26:29.661925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:07:58.920942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643973+0000","last_change":"2024-04-25T01:25:16.743462+0000","last_active":"2024-04-25T01:26:29.643973+0000","last_peered":"2024-04-25T01:26:29.643973+0000","last_clean":"2024-04-25T01:26:29.643973+0000","last_became_active":"2024-04-25T01:25:16.743055+0000","last_became_peered":"2024-04-25T01:25:16.743055+0000","last_unstale":"2024-04-25T01:26:29.643973+0000","last_undegraded":"2024-04-25T01:26:29.643973+0000","last_fullsized":"2024-04-25T01:26:29.643973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:23:03.660065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.641408+0000","last_change":"2024-04-25T01:25:16.751446+0000","last_active":"2024-04-25T01:26:29.641408+0000","last_peered":"2024-04-25T01:26:29.641408+0000","last_clean":"2024-04-25T01:26:29.641408+0000","last_became_active":"2024-04-25T01:25:16.751107+0000","last_became_peered":"2024-04-25T01:25:16.751107+0000","last_unstale":"2024-04-25T01:26:29.641408+0000","last_undegraded":"2024-04-25T01:26:29.641408+0000","last_fullsized":"2024-04-25T01:26:29.641408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:52.275662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654267+0000","last_change":"2024-04-25T01:25:16.748478+0000","last_active":"2024-04-25T01:26:29.654267+0000","last_peered":"2024-04-25T01:26:29.654267+0000","last_clean":"2024-04-25T01:26:29.654267+0000","last_became_active":"2024-04-25T01:25:16.748064+0000","last_became_peered":"2024-04-25T01:25:16.748064+0000","last_unstale":"2024-04-25T01:26:29.654267+0000","last_undegraded":"2024-04-25T01:26:29.654267+0000","last_fullsized":"2024-04-25T01:26:29.654267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:45.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181977+0000","last_change":"2024-04-25T01:25:15.987611+0000","last_active":"2024-04-25T01:26:01.181977+0000","last_peered":"2024-04-25T01:26:01.181977+0000","last_clean":"2024-04-25T01:26:01.181977+0000","last_became_active":"2024-04-25T01:25:15.987300+0000","last_became_peered":"2024-04-25T01:25:15.987300+0000","last_unstale":"2024-04-25T01:26:01.181977+0000","last_undegraded":"2024-04-25T01:26:01.181977+0000","last_fullsized":"2024-04-25T01:26:01.181977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:24.636607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.645469+0000","last_change":"2024-04-25T01:25:16.751076+0000","last_active":"2024-04-25T01:26:29.645469+0000","last_peered":"2024-04-25T01:26:29.645469+0000","last_clean":"2024-04-25T01:26:29.645469+0000","last_became_active":"2024-04-25T01:25:16.750219+0000","last_became_peered":"2024-04-25T01:25:16.750219+0000","last_unstale":"2024-04-25T01:26:29.645469+0000","last_undegraded":"2024-04-25T01:26:29.645469+0000","last_fullsized":"2024-04-25T01:26:29.645469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:11.956004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.182371+0000","last_change":"2024-04-25T01:25:15.977434+0000","last_active":"2024-04-25T01:26:01.182371+0000","last_peered":"2024-04-25T01:26:01.182371+0000","last_clean":"2024-04-25T01:26:01.182371+0000","last_became_active":"2024-04-25T01:25:15.977095+0000","last_became_peered":"2024-04-25T01:25:15.977095+0000","last_unstale":"2024-04-25T01:26:01.182371+0000","last_undegraded":"2024-04-25T01:26:01.182371+0000","last_fullsized":"2024-04-25T01:26:01.182371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:01.102962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658893+0000","last_change":"2024-04-25T01:25:16.740061+0000","last_active":"2024-04-25T01:26:29.658893+0000","last_peered":"2024-04-25T01:26:29.658893+0000","last_clean":"2024-04-25T01:26:29.658893+0000","last_became_active":"2024-04-25T01:25:16.739593+0000","last_became_peered":"2024-04-25T01:25:16.739593+0000","last_unstale":"2024-04-25T01:26:29.658893+0000","last_undegraded":"2024-04-25T01:26:29.658893+0000","last_fullsized":"2024-04-25T01:26:29.658893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:32.711066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.180580+0000","last_change":"2024-04-25T01:25:15.984150+0000","last_active":"2024-04-25T01:26:01.180580+0000","last_peered":"2024-04-25T01:26:01.180580+0000","last_clean":"2024-04-25T01:26:01.180580+0000","last_became_active":"2024-04-25T01:25:15.983405+0000","last_became_peered":"2024-04-25T01:25:15.983405+0000","last_unstale":"2024-04-25T01:26:01.180580+0000","last_undegraded":"2024-04-25T01:26:01.180580+0000","last_fullsized":"2024-04-25T01:26:01.180580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:38.511851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.659171+0000","last_change":"2024-04-25T01:25:16.751549+0000","last_active":"2024-04-25T01:26:29.659171+0000","last_peered":"2024-04-25T01:26:29.659171+0000","last_clean":"2024-04-25T01:26:29.659171+0000","last_became_active":"2024-04-25T01:25:16.751167+0000","last_became_peered":"2024-04-25T01:25:16.751167+0000","last_unstale":"2024-04-25T01:26:29.659171+0000","last_undegraded":"2024-04-25T01:26:29.659171+0000","last_fullsized":"2024-04-25T01:26:29.659171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:21.260019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.179808+0000","last_change":"2024-04-25T01:25:15.977916+0000","last_active":"2024-04-25T01:26:01.179808+0000","last_peered":"2024-04-25T01:26:01.179808+0000","last_clean":"2024-04-25T01:26:01.179808+0000","last_became_active":"2024-04-25T01:25:15.977602+0000","last_became_peered":"2024-04-25T01:25:15.977602+0000","last_unstale":"2024-04-25T01:26:01.179808+0000","last_undegraded":"2024-04-25T01:26:01.179808+0000","last_fullsized":"2024-04-25T01:26:01.179808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:27.926489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.655417+0000","last_change":"2024-04-25T01:25:16.756728+0000","last_active":"2024-04-25T01:26:29.655417+0000","last_peered":"2024-04-25T01:26:29.655417+0000","last_clean":"2024-04-25T01:26:29.655417+0000","last_became_active":"2024-04-25T01:25:16.756363+0000","last_became_peered":"2024-04-25T01:25:16.756363+0000","last_unstale":"2024-04-25T01:26:29.655417+0000","last_undegraded":"2024-04-25T01:26:29.655417+0000","last_fullsized":"2024-04-25T01:26:29.655417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:06.582831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:32.185390+0000","last_change":"2024-04-25T01:25:15.987099+0000","last_active":"2024-04-25T01:26:32.185390+0000","last_peered":"2024-04-25T01:26:32.185390+0000","last_clean":"2024-04-25T01:26:32.185390+0000","last_became_active":"2024-04-25T01:25:15.986745+0000","last_became_peered":"2024-04-25T01:25:15.986745+0000","last_unstale":"2024-04-25T01:26:32.185390+0000","last_undegraded":"2024-04-25T01:26:32.185390+0000","last_fullsized":"2024-04-25T01:26:32.185390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:14.944342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.633696+0000","last_change":"2024-04-25T01:25:16.742735+0000","last_active":"2024-04-25T01:26:29.633696+0000","last_peered":"2024-04-25T01:26:29.633696+0000","last_clean":"2024-04-25T01:26:29.633696+0000","last_became_active":"2024-04-25T01:25:16.742391+0000","last_became_peered":"2024-04-25T01:25:16.742391+0000","last_unstale":"2024-04-25T01:26:29.633696+0000","last_undegraded":"2024-04-25T01:26:29.633696+0000","last_fullsized":"2024-04-25T01:26:29.633696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:05.783111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181320+0000","last_change":"2024-04-25T01:25:15.982605+0000","last_active":"2024-04-25T01:26:01.181320+0000","last_peered":"2024-04-25T01:26:01.181320+0000","last_clean":"2024-04-25T01:26:01.181320+0000","last_became_active":"2024-04-25T01:25:15.982264+0000","last_became_peered":"2024-04-25T01:25:15.982264+0000","last_unstale":"2024-04-25T01:26:01.181320+0000","last_undegraded":"2024-04-25T01:26:01.181320+0000","last_fullsized":"2024-04-25T01:26:01.181320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:09.960827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654239+0000","last_change":"2024-04-25T01:25:16.747784+0000","last_active":"2024-04-25T01:26:29.654239+0000","last_peered":"2024-04-25T01:26:29.654239+0000","last_clean":"2024-04-25T01:26:29.654239+0000","last_became_active":"2024-04-25T01:25:16.747432+0000","last_became_peered":"2024-04-25T01:25:16.747432+0000","last_unstale":"2024-04-25T01:26:29.654239+0000","last_undegraded":"2024-04-25T01:26:29.654239+0000","last_fullsized":"2024-04-25T01:26:29.654239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:40.602582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181269+0000","last_change":"2024-04-25T01:25:15.977370+0000","last_active":"2024-04-25T01:26:01.181269+0000","last_peered":"2024-04-25T01:26:01.181269+0000","last_clean":"2024-04-25T01:26:01.181269+0000","last_became_active":"2024-04-25T01:25:15.977010+0000","last_became_peered":"2024-04-25T01:25:15.977010+0000","last_unstale":"2024-04-25T01:26:01.181269+0000","last_undegraded":"2024-04-25T01:26:01.181269+0000","last_fullsized":"2024-04-25T01:26:01.181269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:41.239154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643866+0000","last_change":"2024-04-25T01:25:16.741490+0000","last_active":"2024-04-25T01:26:29.643866+0000","last_peered":"2024-04-25T01:26:29.643866+0000","last_clean":"2024-04-25T01:26:29.643866+0000","last_became_active":"2024-04-25T01:25:16.741148+0000","last_became_peered":"2024-04-25T01:25:16.741148+0000","last_unstale":"2024-04-25T01:26:29.643866+0000","last_undegraded":"2024-04-25T01:26:29.643866+0000","last_fullsized":"2024-04-25T01:26:29.643866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:50.918575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.192446+0000","last_change":"2024-04-25T01:25:15.988283+0000","last_active":"2024-04-25T01:26:01.192446+0000","last_peered":"2024-04-25T01:26:01.192446+0000","last_clean":"2024-04-25T01:26:01.192446+0000","last_became_active":"2024-04-25T01:25:15.987894+0000","last_became_peered":"2024-04-25T01:25:15.987894+0000","last_unstale":"2024-04-25T01:26:01.192446+0000","last_undegraded":"2024-04-25T01:26:01.192446+0000","last_fullsized":"2024-04-25T01:26:01.192446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:01.220170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661223+0000","last_change":"2024-04-25T01:26:01.287127+0000","last_active":"2024-04-25T01:26:29.661223+0000","last_peered":"2024-04-25T01:26:29.661223+0000","last_clean":"2024-04-25T01:26:29.661223+0000","last_became_active":"2024-04-25T01:26:01.286830+0000","last_became_peered":"2024-04-25T01:26:01.286830+0000","last_unstale":"2024-04-25T01:26:29.661223+0000","last_undegraded":"2024-04-25T01:26:29.661223+0000","last_fullsized":"2024-04-25T01:26:29.661223+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:42:33.695703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:33.186120+0000","last_change":"2024-04-25T01:25:15.981590+0000","last_active":"2024-04-25T01:26:33.186120+0000","last_peered":"2024-04-25T01:26:33.186120+0000","last_clean":"2024-04-25T01:26:33.186120+0000","last_became_active":"2024-04-25T01:25:15.981192+0000","last_became_peered":"2024-04-25T01:25:15.981192+0000","last_unstale":"2024-04-25T01:26:33.186120+0000","last_undegraded":"2024-04-25T01:26:33.186120+0000","last_fullsized":"2024-04-25T01:26:33.186120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:06.220365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658412+0000","last_change":"2024-04-25T01:25:16.751414+0000","last_active":"2024-04-25T01:26:29.658412+0000","last_peered":"2024-04-25T01:26:29.658412+0000","last_clean":"2024-04-25T01:26:29.658412+0000","last_became_active":"2024-04-25T01:25:16.751003+0000","last_became_peered":"2024-04-25T01:25:16.751003+0000","last_unstale":"2024-04-25T01:26:29.658412+0000","last_undegraded":"2024-04-25T01:26:29.658412+0000","last_fullsized":"2024-04-25T01:26:29.658412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:49:03.146685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.286169+0000","last_change":"2024-04-25T01:25:15.974041+0000","last_active":"2024-04-25T01:26:01.286169+0000","last_peered":"2024-04-25T01:26:01.286169+0000","last_clean":"2024-04-25T01:26:01.286169+0000","last_became_active":"2024-04-25T01:25:15.973621+0000","last_became_peered":"2024-04-25T01:25:15.973621+0000","last_unstale":"2024-04-25T01:26:01.286169+0000","last_undegraded":"2024-04-25T01:26:01.286169+0000","last_fullsized":"2024-04-25T01:26:01.286169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:12.039161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.648005+0000","last_change":"2024-04-25T01:25:16.745295+0000","last_active":"2024-04-25T01:26:29.648005+0000","last_peered":"2024-04-25T01:26:29.648005+0000","last_clean":"2024-04-25T01:26:29.648005+0000","last_became_active":"2024-04-25T01:25:16.744931+0000","last_became_peered":"2024-04-25T01:25:16.744931+0000","last_unstale":"2024-04-25T01:26:29.648005+0000","last_undegraded":"2024-04-25T01:26:29.648005+0000","last_fullsized":"2024-04-25T01:26:29.648005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:48.993537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.193101+0000","last_change":"2024-04-25T01:25:15.980900+0000","last_active":"2024-04-25T01:26:01.193101+0000","last_peered":"2024-04-25T01:26:01.193101+0000","last_clean":"2024-04-25T01:26:01.193101+0000","last_became_active":"2024-04-25T01:25:15.980528+0000","last_became_peered":"2024-04-25T01:25:15.980528+0000","last_unstale":"2024-04-25T01:26:01.193101+0000","last_undegraded":"2024-04-25T01:26:01.193101+0000","last_fullsized":"2024-04-25T01:26:01.193101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:47.743151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.660628+0000","last_change":"2024-04-25T01:25:16.753886+0000","last_active":"2024-04-25T01:26:29.660628+0000","last_peered":"2024-04-25T01:26:29.660628+0000","last_clean":"2024-04-25T01:26:29.660628+0000","last_became_active":"2024-04-25T01:25:16.753522+0000","last_became_peered":"2024-04-25T01:25:16.753522+0000","last_unstale":"2024-04-25T01:26:29.660628+0000","last_undegraded":"2024-04-25T01:26:29.660628+0000","last_fullsized":"2024-04-25T01:26:29.660628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:12:46.103249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":225,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:34.238907+0000","last_change":"2024-04-25T01:26:01.194086+0000","last_active":"2024-04-25T01:26:34.238907+0000","last_peered":"2024-04-25T01:26:34.238907+0000","last_clean":"2024-04-25T01:26:34.238907+0000","last_became_active":"2024-04-25T01:26:01.186240+0000","last_became_peered":"2024-04-25T01:26:01.186240+0000","last_unstale":"2024-04-25T01:26:34.238907+0000","last_undegraded":"2024-04-25T01:26:34.238907+0000","last_fullsized":"2024-04-25T01:26:34.238907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:31.758268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":147,"num_read_kb":133,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643176+0000","last_change":"2024-04-25T01:25:16.739726+0000","last_active":"2024-04-25T01:26:29.643176+0000","last_peered":"2024-04-25T01:26:29.643176+0000","last_clean":"2024-04-25T01:26:29.643176+0000","last_became_active":"2024-04-25T01:25:16.739399+0000","last_became_peered":"2024-04-25T01:25:16.739399+0000","last_unstale":"2024-04-25T01:26:29.643176+0000","last_undegraded":"2024-04-25T01:26:29.643176+0000","last_fullsized":"2024-04-25T01:26:29.643176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:21:42.747061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.184400+0000","last_change":"2024-04-25T01:25:15.976731+0000","last_active":"2024-04-25T01:26:01.184400+0000","last_peered":"2024-04-25T01:26:01.184400+0000","last_clean":"2024-04-25T01:26:01.184400+0000","last_became_active":"2024-04-25T01:25:15.976405+0000","last_became_peered":"2024-04-25T01:25:15.976405+0000","last_unstale":"2024-04-25T01:26:01.184400+0000","last_undegraded":"2024-04-25T01:26:01.184400+0000","last_fullsized":"2024-04-25T01:26:01.184400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:21.612389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.659889+0000","last_change":"2024-04-25T01:25:16.741378+0000","last_active":"2024-04-25T01:26:29.659889+0000","last_peered":"2024-04-25T01:26:29.659889+0000","last_clean":"2024-04-25T01:26:29.659889+0000","last_became_active":"2024-04-25T01:25:16.741066+0000","last_became_peered":"2024-04-25T01:25:16.741066+0000","last_unstale":"2024-04-25T01:26:29.659889+0000","last_undegraded":"2024-04-25T01:26:29.659889+0000","last_fullsized":"2024-04-25T01:26:29.659889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:37.288181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.192510+0000","last_change":"2024-04-25T01:25:15.988288+0000","last_active":"2024-04-25T01:26:01.192510+0000","last_peered":"2024-04-25T01:26:01.192510+0000","last_clean":"2024-04-25T01:26:01.192510+0000","last_became_active":"2024-04-25T01:25:15.987860+0000","last_became_peered":"2024-04-25T01:25:15.987860+0000","last_unstale":"2024-04-25T01:26:01.192510+0000","last_undegraded":"2024-04-25T01:26:01.192510+0000","last_fullsized":"2024-04-25T01:26:01.192510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:23.634199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.663569+0000","last_change":"2024-04-25T01:25:16.743784+0000","last_active":"2024-04-25T01:26:29.663569+0000","last_peered":"2024-04-25T01:26:29.663569+0000","last_clean":"2024-04-25T01:26:29.663569+0000","last_became_active":"2024-04-25T01:25:16.743435+0000","last_became_peered":"2024-04-25T01:25:16.743435+0000","last_unstale":"2024-04-25T01:26:29.663569+0000","last_undegraded":"2024-04-25T01:26:29.663569+0000","last_fullsized":"2024-04-25T01:26:29.663569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:52:08.297617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":102,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:33.186829+0000","last_change":"2024-04-25T01:25:15.978201+0000","last_active":"2024-04-25T01:26:33.186829+0000","last_peered":"2024-04-25T01:26:33.186829+0000","last_clean":"2024-04-25T01:26:33.186829+0000","last_became_active":"2024-04-25T01:25:15.977832+0000","last_became_peered":"2024-04-25T01:25:15.977832+0000","last_unstale":"2024-04-25T01:26:33.186829+0000","last_undegraded":"2024-04-25T01:26:33.186829+0000","last_fullsized":"2024-04-25T01:26:33.186829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:42.483969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661981+0000","last_change":"2024-04-25T01:25:16.750944+0000","last_active":"2024-04-25T01:26:29.661981+0000","last_peered":"2024-04-25T01:26:29.661981+0000","last_clean":"2024-04-25T01:26:29.661981+0000","last_became_active":"2024-04-25T01:25:16.750588+0000","last_became_peered":"2024-04-25T01:25:16.750588+0000","last_unstale":"2024-04-25T01:26:29.661981+0000","last_undegraded":"2024-04-25T01:26:29.661981+0000","last_fullsized":"2024-04-25T01:26:29.661981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:31.473184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.202567+0000","last_change":"2024-04-25T01:26:01.202137+0000","last_active":"2024-04-25T01:26:01.202567+0000","last_peered":"2024-04-25T01:26:01.202567+0000","last_clean":"2024-04-25T01:26:01.202567+0000","last_became_active":"2024-04-25T01:26:01.186142+0000","last_became_peered":"2024-04-25T01:26:01.186142+0000","last_unstale":"2024-04-25T01:26:01.202567+0000","last_undegraded":"2024-04-25T01:26:01.202567+0000","last_fullsized":"2024-04-25T01:26:01.202567+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:36.628455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.653831+0000","last_change":"2024-04-25T01:25:16.748178+0000","last_active":"2024-04-25T01:26:29.653831+0000","last_peered":"2024-04-25T01:26:29.653831+0000","last_clean":"2024-04-25T01:26:29.653831+0000","last_became_active":"2024-04-25T01:25:16.747868+0000","last_became_peered":"2024-04-25T01:25:16.747868+0000","last_unstale":"2024-04-25T01:26:29.653831+0000","last_undegraded":"2024-04-25T01:26:29.653831+0000","last_fullsized":"2024-04-25T01:26:29.653831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:06.378387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.179808+0000","last_change":"2024-04-25T01:25:15.981651+0000","last_active":"2024-04-25T01:26:01.179808+0000","last_peered":"2024-04-25T01:26:01.179808+0000","last_clean":"2024-04-25T01:26:01.179808+0000","last_became_active":"2024-04-25T01:25:15.981245+0000","last_became_peered":"2024-04-25T01:25:15.981245+0000","last_unstale":"2024-04-25T01:26:01.179808+0000","last_undegraded":"2024-04-25T01:26:01.179808+0000","last_fullsized":"2024-04-25T01:26:01.179808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:20.009347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.628024+0000","last_change":"2024-04-25T01:25:16.756873+0000","last_active":"2024-04-25T01:26:29.628024+0000","last_peered":"2024-04-25T01:26:29.628024+0000","last_clean":"2024-04-25T01:26:29.628024+0000","last_became_active":"2024-04-25T01:25:16.756505+0000","last_became_peered":"2024-04-25T01:25:16.756505+0000","last_unstale":"2024-04-25T01:26:29.628024+0000","last_undegraded":"2024-04-25T01:26:29.628024+0000","last_fullsized":"2024-04-25T01:26:29.628024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:23.412303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181032+0000","last_change":"2024-04-25T01:25:15.983640+0000","last_active":"2024-04-25T01:26:01.181032+0000","last_peered":"2024-04-25T01:26:01.181032+0000","last_clean":"2024-04-25T01:26:01.181032+0000","last_became_active":"2024-04-25T01:25:15.983280+0000","last_became_peered":"2024-04-25T01:25:15.983280+0000","last_unstale":"2024-04-25T01:26:01.181032+0000","last_undegraded":"2024-04-25T01:26:01.181032+0000","last_fullsized":"2024-04-25T01:26:01.181032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:18.493139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643060+0000","last_change":"2024-04-25T01:25:16.751616+0000","last_active":"2024-04-25T01:26:29.643060+0000","last_peered":"2024-04-25T01:26:29.643060+0000","last_clean":"2024-04-25T01:26:29.643060+0000","last_became_active":"2024-04-25T01:25:16.751254+0000","last_became_peered":"2024-04-25T01:25:16.751254+0000","last_unstale":"2024-04-25T01:26:29.643060+0000","last_undegraded":"2024-04-25T01:26:29.643060+0000","last_fullsized":"2024-04-25T01:26:29.643060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:55.556054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.286025+0000","last_change":"2024-04-25T01:25:15.977774+0000","last_active":"2024-04-25T01:26:01.286025+0000","last_peered":"2024-04-25T01:26:01.286025+0000","last_clean":"2024-04-25T01:26:01.286025+0000","last_became_active":"2024-04-25T01:25:15.977398+0000","last_became_peered":"2024-04-25T01:25:15.977398+0000","last_unstale":"2024-04-25T01:26:01.286025+0000","last_undegraded":"2024-04-25T01:26:01.286025+0000","last_fullsized":"2024-04-25T01:26:01.286025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:33.729091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.645162+0000","last_change":"2024-04-25T01:26:01.185468+0000","last_active":"2024-04-25T01:26:29.645162+0000","last_peered":"2024-04-25T01:26:29.645162+0000","last_clean":"2024-04-25T01:26:29.645162+0000","last_became_active":"2024-04-25T01:26:01.184888+0000","last_became_peered":"2024-04-25T01:26:01.184888+0000","last_unstale":"2024-04-25T01:26:29.645162+0000","last_undegraded":"2024-04-25T01:26:29.645162+0000","last_fullsized":"2024-04-25T01:26:29.645162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:59.565972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.653980+0000","last_change":"2024-04-25T01:25:16.746447+0000","last_active":"2024-04-25T01:26:29.653980+0000","last_peered":"2024-04-25T01:26:29.653980+0000","last_clean":"2024-04-25T01:26:29.653980+0000","last_became_active":"2024-04-25T01:25:16.745978+0000","last_became_peered":"2024-04-25T01:25:16.745978+0000","last_unstale":"2024-04-25T01:26:29.653980+0000","last_undegraded":"2024-04-25T01:26:29.653980+0000","last_fullsized":"2024-04-25T01:26:29.653980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:41.249662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:30.180738+0000","last_change":"2024-04-25T01:25:15.987083+0000","last_active":"2024-04-25T01:26:30.180738+0000","last_peered":"2024-04-25T01:26:30.180738+0000","last_clean":"2024-04-25T01:26:30.180738+0000","last_became_active":"2024-04-25T01:25:15.986632+0000","last_became_peered":"2024-04-25T01:25:15.986632+0000","last_unstale":"2024-04-25T01:26:30.180738+0000","last_undegraded":"2024-04-25T01:26:30.180738+0000","last_fullsized":"2024-04-25T01:26:30.180738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:44.792239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.637980+0000","last_change":"2024-04-25T01:25:16.742007+0000","last_active":"2024-04-25T01:26:29.637980+0000","last_peered":"2024-04-25T01:26:29.637980+0000","last_clean":"2024-04-25T01:26:29.637980+0000","last_became_active":"2024-04-25T01:25:16.741464+0000","last_became_peered":"2024-04-25T01:25:16.741464+0000","last_unstale":"2024-04-25T01:26:29.637980+0000","last_undegraded":"2024-04-25T01:26:29.637980+0000","last_fullsized":"2024-04-25T01:26:29.637980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:32.171648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181599+0000","last_change":"2024-04-25T01:25:15.978255+0000","last_active":"2024-04-25T01:26:01.181599+0000","last_peered":"2024-04-25T01:26:01.181599+0000","last_clean":"2024-04-25T01:26:01.181599+0000","last_became_active":"2024-04-25T01:25:15.977860+0000","last_became_peered":"2024-04-25T01:25:15.977860+0000","last_unstale":"2024-04-25T01:26:01.181599+0000","last_undegraded":"2024-04-25T01:26:01.181599+0000","last_fullsized":"2024-04-25T01:26:01.181599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.516137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.285521+0000","last_change":"2024-04-25T01:23:58.204099+0000","last_active":"2024-04-25T01:26:01.285521+0000","last_peered":"2024-04-25T01:26:01.285521+0000","last_clean":"2024-04-25T01:26:01.285521+0000","last_became_active":"2024-04-25T01:23:58.203533+0000","last_became_peered":"2024-04-25T01:23:58.203533+0000","last_unstale":"2024-04-25T01:26:01.285521+0000","last_undegraded":"2024-04-25T01:26:01.285521+0000","last_fullsized":"2024-04-25T01:26:01.285521+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_clean_scrub_stamp":"2024-04-25T01:20:28.584758+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:17.504360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.653755+0000","last_change":"2024-04-25T01:26:01.186051+0000","last_active":"2024-04-25T01:26:29.653755+0000","last_peered":"2024-04-25T01:26:29.653755+0000","last_clean":"2024-04-25T01:26:29.653755+0000","last_became_active":"2024-04-25T01:26:01.185684+0000","last_became_peered":"2024-04-25T01:26:01.185684+0000","last_unstale":"2024-04-25T01:26:29.653755+0000","last_undegraded":"2024-04-25T01:26:29.653755+0000","last_fullsized":"2024-04-25T01:26:29.653755+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:46.754769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.285610+0000","last_change":"2024-04-25T01:25:15.986400+0000","last_active":"2024-04-25T01:26:01.285610+0000","last_peered":"2024-04-25T01:26:01.285610+0000","last_clean":"2024-04-25T01:26:01.285610+0000","last_became_active":"2024-04-25T01:25:15.985975+0000","last_became_peered":"2024-04-25T01:25:15.985975+0000","last_unstale":"2024-04-25T01:26:01.285610+0000","last_undegraded":"2024-04-25T01:26:01.285610+0000","last_fullsized":"2024-04-25T01:26:01.285610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:12.666394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.180431+0000","last_change":"2024-04-25T01:25:15.981370+0000","last_active":"2024-04-25T01:26:01.180431+0000","last_peered":"2024-04-25T01:26:01.180431+0000","last_clean":"2024-04-25T01:26:01.180431+0000","last_became_active":"2024-04-25T01:25:15.980995+0000","last_became_peered":"2024-04-25T01:25:15.980995+0000","last_unstale":"2024-04-25T01:26:01.180431+0000","last_undegraded":"2024-04-25T01:26:01.180431+0000","last_fullsized":"2024-04-25T01:26:01.180431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:29.754441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654232+0000","last_change":"2024-04-25T01:25:16.744887+0000","last_active":"2024-04-25T01:26:29.654232+0000","last_peered":"2024-04-25T01:26:29.654232+0000","last_clean":"2024-04-25T01:26:29.654232+0000","last_became_active":"2024-04-25T01:25:16.744538+0000","last_became_peered":"2024-04-25T01:25:16.744538+0000","last_unstale":"2024-04-25T01:26:29.654232+0000","last_undegraded":"2024-04-25T01:26:29.654232+0000","last_fullsized":"2024-04-25T01:26:29.654232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:32.132521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.180875+0000","last_change":"2024-04-25T01:25:15.990181+0000","last_active":"2024-04-25T01:26:01.180875+0000","last_peered":"2024-04-25T01:26:01.180875+0000","last_clean":"2024-04-25T01:26:01.180875+0000","last_became_active":"2024-04-25T01:25:15.989816+0000","last_became_peered":"2024-04-25T01:25:15.989816+0000","last_unstale":"2024-04-25T01:26:01.180875+0000","last_undegraded":"2024-04-25T01:26:01.180875+0000","last_fullsized":"2024-04-25T01:26:01.180875+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:17:58.377057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.651935+0000","last_change":"2024-04-25T01:25:16.746199+0000","last_active":"2024-04-25T01:26:29.651935+0000","last_peered":"2024-04-25T01:26:29.651935+0000","last_clean":"2024-04-25T01:26:29.651935+0000","last_became_active":"2024-04-25T01:25:16.745857+0000","last_became_peered":"2024-04-25T01:25:16.745857+0000","last_unstale":"2024-04-25T01:26:29.651935+0000","last_undegraded":"2024-04-25T01:26:29.651935+0000","last_fullsized":"2024-04-25T01:26:29.651935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:21.392309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.182299+0000","last_change":"2024-04-25T01:25:15.981173+0000","last_active":"2024-04-25T01:26:01.182299+0000","last_peered":"2024-04-25T01:26:01.182299+0000","last_clean":"2024-04-25T01:26:01.182299+0000","last_became_active":"2024-04-25T01:25:15.980718+0000","last_became_peered":"2024-04-25T01:25:15.980718+0000","last_unstale":"2024-04-25T01:26:01.182299+0000","last_undegraded":"2024-04-25T01:26:01.182299+0000","last_fullsized":"2024-04-25T01:26:01.182299+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:54.086543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.649554+0000","last_change":"2024-04-25T01:25:16.753987+0000","last_active":"2024-04-25T01:26:29.649554+0000","last_peered":"2024-04-25T01:26:29.649554+0000","last_clean":"2024-04-25T01:26:29.649554+0000","last_became_active":"2024-04-25T01:25:16.753546+0000","last_became_peered":"2024-04-25T01:25:16.753546+0000","last_unstale":"2024-04-25T01:26:29.649554+0000","last_undegraded":"2024-04-25T01:26:29.649554+0000","last_fullsized":"2024-04-25T01:26:29.649554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:10.318501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.184414+0000","last_change":"2024-04-25T01:25:15.980938+0000","last_active":"2024-04-25T01:26:01.184414+0000","last_peered":"2024-04-25T01:26:01.184414+0000","last_clean":"2024-04-25T01:26:01.184414+0000","last_became_active":"2024-04-25T01:25:15.980586+0000","last_became_peered":"2024-04-25T01:25:15.980586+0000","last_unstale":"2024-04-25T01:26:01.184414+0000","last_undegraded":"2024-04-25T01:26:01.184414+0000","last_fullsized":"2024-04-25T01:26:01.184414+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:00.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.605599+0000","last_change":"2024-04-25T01:25:16.749711+0000","last_active":"2024-04-25T01:26:29.605599+0000","last_peered":"2024-04-25T01:26:29.605599+0000","last_clean":"2024-04-25T01:26:29.605599+0000","last_became_active":"2024-04-25T01:25:16.749218+0000","last_became_peered":"2024-04-25T01:25:16.749218+0000","last_unstale":"2024-04-25T01:26:29.605599+0000","last_undegraded":"2024-04-25T01:26:29.605599+0000","last_fullsized":"2024-04-25T01:26:29.605599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:56.555204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181476+0000","last_change":"2024-04-25T01:25:15.988257+0000","last_active":"2024-04-25T01:26:01.181476+0000","last_peered":"2024-04-25T01:26:01.181476+0000","last_clean":"2024-04-25T01:26:01.181476+0000","last_became_active":"2024-04-25T01:25:15.987469+0000","last_became_peered":"2024-04-25T01:25:15.987469+0000","last_unstale":"2024-04-25T01:26:01.181476+0000","last_undegraded":"2024-04-25T01:26:01.181476+0000","last_fullsized":"2024-04-25T01:26:01.181476+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:29.849868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.648521+0000","last_change":"2024-04-25T01:25:16.746686+0000","last_active":"2024-04-25T01:26:29.648521+0000","last_peered":"2024-04-25T01:26:29.648521+0000","last_clean":"2024-04-25T01:26:29.648521+0000","last_became_active":"2024-04-25T01:25:16.746028+0000","last_became_peered":"2024-04-25T01:25:16.746028+0000","last_unstale":"2024-04-25T01:26:29.648521+0000","last_undegraded":"2024-04-25T01:26:29.648521+0000","last_fullsized":"2024-04-25T01:26:29.648521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:34:35.172207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.189978+0000","last_change":"2024-04-25T01:26:01.189508+0000","last_active":"2024-04-25T01:26:01.189978+0000","last_peered":"2024-04-25T01:26:01.189978+0000","last_clean":"2024-04-25T01:26:01.189978+0000","last_became_active":"2024-04-25T01:26:01.184031+0000","last_became_peered":"2024-04-25T01:26:01.184031+0000","last_unstale":"2024-04-25T01:26:01.189978+0000","last_undegraded":"2024-04-25T01:26:01.189978+0000","last_fullsized":"2024-04-25T01:26:01.189978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:50.354208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.651185+0000","last_change":"2024-04-25T01:25:16.752510+0000","last_active":"2024-04-25T01:26:29.651185+0000","last_peered":"2024-04-25T01:26:29.651185+0000","last_clean":"2024-04-25T01:26:29.651185+0000","last_became_active":"2024-04-25T01:25:16.751889+0000","last_became_peered":"2024-04-25T01:25:16.751889+0000","last_unstale":"2024-04-25T01:26:29.651185+0000","last_undegraded":"2024-04-25T01:26:29.651185+0000","last_fullsized":"2024-04-25T01:26:29.651185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:44.628980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.179564+0000","last_change":"2024-04-25T01:25:15.983638+0000","last_active":"2024-04-25T01:26:01.179564+0000","last_peered":"2024-04-25T01:26:01.179564+0000","last_clean":"2024-04-25T01:26:01.179564+0000","last_became_active":"2024-04-25T01:25:15.983293+0000","last_became_peered":"2024-04-25T01:25:15.983293+0000","last_unstale":"2024-04-25T01:26:01.179564+0000","last_undegraded":"2024-04-25T01:26:01.179564+0000","last_fullsized":"2024-04-25T01:26:01.179564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:04:27.495446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.659977+0000","last_change":"2024-04-25T01:25:16.736420+0000","last_active":"2024-04-25T01:26:29.659977+0000","last_peered":"2024-04-25T01:26:29.659977+0000","last_clean":"2024-04-25T01:26:29.659977+0000","last_became_active":"2024-04-25T01:25:16.736031+0000","last_became_peered":"2024-04-25T01:25:16.736031+0000","last_unstale":"2024-04-25T01:26:29.659977+0000","last_undegraded":"2024-04-25T01:26:29.659977+0000","last_fullsized":"2024-04-25T01:26:29.659977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:33.380905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.185349+0000","last_change":"2024-04-25T01:25:15.982201+0000","last_active":"2024-04-25T01:26:01.185349+0000","last_peered":"2024-04-25T01:26:01.185349+0000","last_clean":"2024-04-25T01:26:01.185349+0000","last_became_active":"2024-04-25T01:25:15.981820+0000","last_became_peered":"2024-04-25T01:25:15.981820+0000","last_unstale":"2024-04-25T01:26:01.185349+0000","last_undegraded":"2024-04-25T01:26:01.185349+0000","last_fullsized":"2024-04-25T01:26:01.185349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:12.149417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.660402+0000","last_change":"2024-04-25T01:25:16.739126+0000","last_active":"2024-04-25T01:26:29.660402+0000","last_peered":"2024-04-25T01:26:29.660402+0000","last_clean":"2024-04-25T01:26:29.660402+0000","last_became_active":"2024-04-25T01:25:16.738762+0000","last_became_peered":"2024-04-25T01:25:16.738762+0000","last_unstale":"2024-04-25T01:26:29.660402+0000","last_undegraded":"2024-04-25T01:26:29.660402+0000","last_fullsized":"2024-04-25T01:26:29.660402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:00:47.661399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.192997+0000","last_change":"2024-04-25T01:25:15.985732+0000","last_active":"2024-04-25T01:26:01.192997+0000","last_peered":"2024-04-25T01:26:01.192997+0000","last_clean":"2024-04-25T01:26:01.192997+0000","last_became_active":"2024-04-25T01:25:15.985300+0000","last_became_peered":"2024-04-25T01:25:15.985300+0000","last_unstale":"2024-04-25T01:26:01.192997+0000","last_undegraded":"2024-04-25T01:26:01.192997+0000","last_fullsized":"2024-04-25T01:26:01.192997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:01.180165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.657443+0000","last_change":"2024-04-25T01:25:16.746639+0000","last_active":"2024-04-25T01:26:29.657443+0000","last_peered":"2024-04-25T01:26:29.657443+0000","last_clean":"2024-04-25T01:26:29.657443+0000","last_became_active":"2024-04-25T01:25:16.746291+0000","last_became_peered":"2024-04-25T01:25:16.746291+0000","last_unstale":"2024-04-25T01:26:29.657443+0000","last_undegraded":"2024-04-25T01:26:29.657443+0000","last_fullsized":"2024-04-25T01:26:29.657443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:31.346243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181626+0000","last_change":"2024-04-25T01:25:15.989193+0000","last_active":"2024-04-25T01:26:01.181626+0000","last_peered":"2024-04-25T01:26:01.181626+0000","last_clean":"2024-04-25T01:26:01.181626+0000","last_became_active":"2024-04-25T01:25:15.988811+0000","last_became_peered":"2024-04-25T01:25:15.988811+0000","last_unstale":"2024-04-25T01:26:01.181626+0000","last_undegraded":"2024-04-25T01:26:01.181626+0000","last_fullsized":"2024-04-25T01:26:01.181626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:56.030398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.656944+0000","last_change":"2024-04-25T01:25:16.742209+0000","last_active":"2024-04-25T01:26:29.656944+0000","last_peered":"2024-04-25T01:26:29.656944+0000","last_clean":"2024-04-25T01:26:29.656944+0000","last_became_active":"2024-04-25T01:25:16.741810+0000","last_became_peered":"2024-04-25T01:25:16.741810+0000","last_unstale":"2024-04-25T01:26:29.656944+0000","last_undegraded":"2024-04-25T01:26:29.656944+0000","last_fullsized":"2024-04-25T01:26:29.656944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:38.008108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.648828+0000","last_change":"2024-04-25T01:25:16.747264+0000","last_active":"2024-04-25T01:26:29.648828+0000","last_peered":"2024-04-25T01:26:29.648828+0000","last_clean":"2024-04-25T01:26:29.648828+0000","last_became_active":"2024-04-25T01:25:16.746514+0000","last_became_peered":"2024-04-25T01:25:16.746514+0000","last_unstale":"2024-04-25T01:26:29.648828+0000","last_undegraded":"2024-04-25T01:26:29.648828+0000","last_fullsized":"2024-04-25T01:26:29.648828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:14.672062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.646761+0000","last_change":"2024-04-25T01:26:01.194587+0000","last_active":"2024-04-25T01:26:29.646761+0000","last_peered":"2024-04-25T01:26:29.646761+0000","last_clean":"2024-04-25T01:26:29.646761+0000","last_became_active":"2024-04-25T01:26:01.194035+0000","last_became_peered":"2024-04-25T01:26:01.194035+0000","last_unstale":"2024-04-25T01:26:29.646761+0000","last_undegraded":"2024-04-25T01:26:29.646761+0000","last_fullsized":"2024-04-25T01:26:29.646761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:29.304743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.644292+0000","last_change":"2024-04-25T01:25:16.752000+0000","last_active":"2024-04-25T01:26:29.644292+0000","last_peered":"2024-04-25T01:26:29.644292+0000","last_clean":"2024-04-25T01:26:29.644292+0000","last_became_active":"2024-04-25T01:25:16.749443+0000","last_became_peered":"2024-04-25T01:25:16.749443+0000","last_unstale":"2024-04-25T01:26:29.644292+0000","last_undegraded":"2024-04-25T01:26:29.644292+0000","last_fullsized":"2024-04-25T01:26:29.644292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:02.287719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643640+0000","last_change":"2024-04-25T01:25:16.742510+0000","last_active":"2024-04-25T01:26:29.643640+0000","last_peered":"2024-04-25T01:26:29.643640+0000","last_clean":"2024-04-25T01:26:29.643640+0000","last_became_active":"2024-04-25T01:25:16.742162+0000","last_became_peered":"2024-04-25T01:25:16.742162+0000","last_unstale":"2024-04-25T01:26:29.643640+0000","last_undegraded":"2024-04-25T01:26:29.643640+0000","last_fullsized":"2024-04-25T01:26:29.643640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:02.924505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654633+0000","last_change":"2024-04-25T01:25:16.756224+0000","last_active":"2024-04-25T01:26:29.654633+0000","last_peered":"2024-04-25T01:26:29.654633+0000","last_clean":"2024-04-25T01:26:29.654633+0000","last_became_active":"2024-04-25T01:25:16.755868+0000","last_became_peered":"2024-04-25T01:25:16.755868+0000","last_unstale":"2024-04-25T01:26:29.654633+0000","last_undegraded":"2024-04-25T01:26:29.654633+0000","last_fullsized":"2024-04-25T01:26:29.654633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:53.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.637047+0000","last_change":"2024-04-25T01:26:01.184911+0000","last_active":"2024-04-25T01:26:29.637047+0000","last_peered":"2024-04-25T01:26:29.637047+0000","last_clean":"2024-04-25T01:26:29.637047+0000","last_became_active":"2024-04-25T01:26:01.184299+0000","last_became_peered":"2024-04-25T01:26:01.184299+0000","last_unstale":"2024-04-25T01:26:29.637047+0000","last_undegraded":"2024-04-25T01:26:29.637047+0000","last_fullsized":"2024-04-25T01:26:29.637047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:32.156598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.649699+0000","last_change":"2024-04-25T01:25:16.745294+0000","last_active":"2024-04-25T01:26:29.649699+0000","last_peered":"2024-04-25T01:26:29.649699+0000","last_clean":"2024-04-25T01:26:29.649699+0000","last_became_active":"2024-04-25T01:25:16.743285+0000","last_became_peered":"2024-04-25T01:25:16.743285+0000","last_unstale":"2024-04-25T01:26:29.649699+0000","last_undegraded":"2024-04-25T01:26:29.649699+0000","last_fullsized":"2024-04-25T01:26:29.649699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:36.189339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.638875+0000","last_change":"2024-04-25T01:26:01.185479+0000","last_active":"2024-04-25T01:26:29.638875+0000","last_peered":"2024-04-25T01:26:29.638875+0000","last_clean":"2024-04-25T01:26:29.638875+0000","last_became_active":"2024-04-25T01:26:01.185073+0000","last_became_peered":"2024-04-25T01:26:01.185073+0000","last_unstale":"2024-04-25T01:26:29.638875+0000","last_undegraded":"2024-04-25T01:26:29.638875+0000","last_fullsized":"2024-04-25T01:26:29.638875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:32.184596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.635518+0000","last_change":"2024-04-25T01:26:01.184323+0000","last_active":"2024-04-25T01:26:29.635518+0000","last_peered":"2024-04-25T01:26:29.635518+0000","last_clean":"2024-04-25T01:26:29.635518+0000","last_became_active":"2024-04-25T01:26:01.183875+0000","last_became_peered":"2024-04-25T01:26:01.183875+0000","last_unstale":"2024-04-25T01:26:29.635518+0000","last_undegraded":"2024-04-25T01:26:29.635518+0000","last_fullsized":"2024-04-25T01:26:29.635518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:54.475788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.634802+0000","last_change":"2024-04-25T01:25:16.750026+0000","last_active":"2024-04-25T01:26:29.634802+0000","last_peered":"2024-04-25T01:26:29.634802+0000","last_clean":"2024-04-25T01:26:29.634802+0000","last_became_active":"2024-04-25T01:25:16.749588+0000","last_became_peered":"2024-04-25T01:25:16.749588+0000","last_unstale":"2024-04-25T01:26:29.634802+0000","last_undegraded":"2024-04-25T01:26:29.634802+0000","last_fullsized":"2024-04-25T01:26:29.634802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:09.357719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.662830+0000","last_change":"2024-04-25T01:25:16.743440+0000","last_active":"2024-04-25T01:26:29.662830+0000","last_peered":"2024-04-25T01:26:29.662830+0000","last_clean":"2024-04-25T01:26:29.662830+0000","last_became_active":"2024-04-25T01:25:16.742904+0000","last_became_peered":"2024-04-25T01:25:16.742904+0000","last_unstale":"2024-04-25T01:26:29.662830+0000","last_undegraded":"2024-04-25T01:26:29.662830+0000","last_fullsized":"2024-04-25T01:26:29.662830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:17.380260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661691+0000","last_change":"2024-04-25T01:25:16.738305+0000","last_active":"2024-04-25T01:26:29.661691+0000","last_peered":"2024-04-25T01:26:29.661691+0000","last_clean":"2024-04-25T01:26:29.661691+0000","last_became_active":"2024-04-25T01:25:16.737907+0000","last_became_peered":"2024-04-25T01:25:16.737907+0000","last_unstale":"2024-04-25T01:26:29.661691+0000","last_undegraded":"2024-04-25T01:26:29.661691+0000","last_fullsized":"2024-04-25T01:26:29.661691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:29.501388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.633237+0000","last_change":"2024-04-25T01:25:16.757755+0000","last_active":"2024-04-25T01:26:29.633237+0000","last_peered":"2024-04-25T01:26:29.633237+0000","last_clean":"2024-04-25T01:26:29.633237+0000","last_became_active":"2024-04-25T01:25:16.757374+0000","last_became_peered":"2024-04-25T01:25:16.757374+0000","last_unstale":"2024-04-25T01:26:29.633237+0000","last_undegraded":"2024-04-25T01:26:29.633237+0000","last_fullsized":"2024-04-25T01:26:29.633237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:46.848407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250979,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":458,"num_read_kb":297,"num_write":163,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":61158,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":196311,"data_compressed":13269,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":456499,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":1109695,"data_compressed":44224,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456623,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456431,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1048614,"data_compressed":42368,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456012,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":517100,"data_compressed":7667,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":457283,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":457155,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":456323,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":518276,"data_compressed":15368,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1047766,"data_compressed":42368,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":62471,"data_compressed":9557,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63405,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:26:38.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:26:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:38.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:38 smithi016 ceph-mon[29153]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:26:38.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:38 smithi038 ceph-mon[32841]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:26:38.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:38 smithi156 ceph-mon[32631]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:26:39.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:39 smithi016 ceph-mon[29153]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:26:39.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:39 smithi038 ceph-mon[32841]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:26:39.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:39 smithi156 ceph-mon[32631]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:26:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 -- 172.21.15.16:0/2921512669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c069f40 msgr2=0x7fae3c06a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 --2- 172.21.15.16:0/2921512669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c069f40 0x7fae3c06a3a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009990 tx=0x7fae2c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 -- 172.21.15.16:0/2921512669 shutdown_connections 2024-04-25T01:26:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 --2- 172.21.15.16:0/2921512669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 0x7fae3c10f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 --2- 172.21.15.16:0/2921512669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae3c1082b0 0x7fae3c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 --2- 172.21.15.16:0/2921512669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c069f40 0x7fae3c06a3a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 -- 172.21.15.16:0/2921512669 >> 172.21.15.16:0/2921512669 conn(0x7fae3c0780e0 msgr2=0x7fae3c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 -- 172.21.15.16:0/2921512669 shutdown_connections 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.274+0000 7fae43773700 1 -- 172.21.15.16:0/2921512669 wait complete. 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 Processor -- start 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 -- start start 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae3c069f40 0x7fae3c101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 0x7fae3c101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c1082b0 0x7fae3c101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c112440 con 0x7fae3c1082b0 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1122c0 con 0x7fae3c069f40 2024-04-25T01:26:40.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae43773700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1125c0 con 0x7fae3c06a8e0 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae40d0e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 0x7fae3c101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae40d0e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 0x7fae3c101740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44774/0 (socket says 172.21.15.16:44774) 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae41d10700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c1082b0 0x7fae3c101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae40d0e700 1 -- 172.21.15.16:0/2196842558 learned_addr learned my addr 172.21.15.16:0/2196842558 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.275+0000 7fae4150f700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae3c069f40 0x7fae3c101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae41d10700 1 -- 172.21.15.16:0/2196842558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 msgr2=0x7fae3c101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae41d10700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 0x7fae3c101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae41d10700 1 -- 172.21.15.16:0/2196842558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae3c069f40 msgr2=0x7fae3c101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae41d10700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae3c069f40 0x7fae3c101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae41d10700 1 -- 172.21.15.16:0/2196842558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3c1a6f60 con 0x7fae3c1082b0 2024-04-25T01:26:40.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae41d10700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c1082b0 0x7fae3c101c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c7a0 tx=0x7fae3400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:40.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34004a30 con 0x7fae3c1082b0 2024-04-25T01:26:40.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae340054d0 con 0x7fae3c1082b0 2024-04-25T01:26:40.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3c1a7250 con 0x7fae3c1082b0 2024-04-25T01:26:40.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3400ec10 con 0x7fae3c1082b0 2024-04-25T01:26:40.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.276+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae3c1a77e0 con 0x7fae3c1082b0 2024-04-25T01:26:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.277+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae3c10a820 con 0x7fae3c1082b0 2024-04-25T01:26:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.278+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fae34005020 con 0x7fae3c1082b0 2024-04-25T01:26:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.279+0000 7fae327fc700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fae280a2ca0 0x7fae280a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.279+0000 7fae4150f700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fae280a2ca0 0x7fae280a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.279+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fae340ceaf0 con 0x7fae3c1082b0 2024-04-25T01:26:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.279+0000 7fae4150f700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fae280a2ca0 0x7fae280a5160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fae3800a400 tx=0x7fae3800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:40.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.283+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fae3409bac0 con 0x7fae3c1082b0 2024-04-25T01:26:40.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.485+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae3c102730 con 0x7fae280a2ca0 2024-04-25T01:26:40.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.492+0000 7fae327fc700 1 -- 172.21.15.16:0/2196842558 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259131 (secure 0 0 0) 0x7fae3c102730 con 0x7fae280a2ca0 2024-04-25T01:26:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fae280a2ca0 msgr2=0x7fae280a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fae280a2ca0 0x7fae280a5160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fae3800a400 tx=0x7fae3800d040 comp rx=0 tx=0).stop 2024-04-25T01:26:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c1082b0 msgr2=0x7fae3c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c1082b0 0x7fae3c101c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c7a0 tx=0x7fae3400cab0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 shutdown_connections 2024-04-25T01:26:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fae280a2ca0 0x7fae280a5160 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae3c1082b0 0x7fae3c101c80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae3c069f40 0x7fae3c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 --2- 172.21.15.16:0/2196842558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae3c06a8e0 0x7fae3c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 >> 172.21.15.16:0/2196842558 conn(0x7fae3c0780e0 msgr2=0x7fae3c0fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 shutdown_connections 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:40.495+0000 7fae43773700 1 -- 172.21.15.16:0/2196842558 wait complete. 2024-04-25T01:26:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-25T01:26:41.171 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-04-25T01:26:39.630142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":594,"num_read_kb":411,"num_write":346,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1241894,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":780,"ondisk_log_size":780,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368988,"kb_used_data":11516,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572836,"statfs":{"total":1151940427776,"available":1151562584064,"internally_reserved":0,"allocated":11792384,"data_stored":7437777,"data_compressed":192672,"data_compressed_allocated":253952,"data_compressed_original":2039808,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":32,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643363+0000","last_change":"2024-04-25T01:25:16.740766+0000","last_active":"2024-04-25T01:26:29.643363+0000","last_peered":"2024-04-25T01:26:29.643363+0000","last_clean":"2024-04-25T01:26:29.643363+0000","last_became_active":"2024-04-25T01:25:16.740181+0000","last_became_peered":"2024-04-25T01:25:16.740181+0000","last_unstale":"2024-04-25T01:26:29.643363+0000","last_undegraded":"2024-04-25T01:26:29.643363+0000","last_fullsized":"2024-04-25T01:26:29.643363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:59.154389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.645118+0000","last_change":"2024-04-25T01:25:16.750724+0000","last_active":"2024-04-25T01:26:29.645118+0000","last_peered":"2024-04-25T01:26:29.645118+0000","last_clean":"2024-04-25T01:26:29.645118+0000","last_became_active":"2024-04-25T01:25:16.750391+0000","last_became_peered":"2024-04-25T01:25:16.750391+0000","last_unstale":"2024-04-25T01:26:29.645118+0000","last_undegraded":"2024-04-25T01:26:29.645118+0000","last_fullsized":"2024-04-25T01:26:29.645118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:15.576020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658916+0000","last_change":"2024-04-25T01:25:16.753914+0000","last_active":"2024-04-25T01:26:29.658916+0000","last_peered":"2024-04-25T01:26:29.658916+0000","last_clean":"2024-04-25T01:26:29.658916+0000","last_became_active":"2024-04-25T01:25:16.753464+0000","last_became_peered":"2024-04-25T01:25:16.753464+0000","last_unstale":"2024-04-25T01:26:29.658916+0000","last_undegraded":"2024-04-25T01:26:29.658916+0000","last_fullsized":"2024-04-25T01:26:29.658916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:55.716948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.655867+0000","last_change":"2024-04-25T01:25:16.754586+0000","last_active":"2024-04-25T01:26:29.655867+0000","last_peered":"2024-04-25T01:26:29.655867+0000","last_clean":"2024-04-25T01:26:29.655867+0000","last_became_active":"2024-04-25T01:25:16.752260+0000","last_became_peered":"2024-04-25T01:25:16.752260+0000","last_unstale":"2024-04-25T01:26:29.655867+0000","last_undegraded":"2024-04-25T01:26:29.655867+0000","last_fullsized":"2024-04-25T01:26:29.655867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:30.935184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.629434+0000","last_change":"2024-04-25T01:25:16.741397+0000","last_active":"2024-04-25T01:26:29.629434+0000","last_peered":"2024-04-25T01:26:29.629434+0000","last_clean":"2024-04-25T01:26:29.629434+0000","last_became_active":"2024-04-25T01:25:16.741056+0000","last_became_peered":"2024-04-25T01:25:16.741056+0000","last_unstale":"2024-04-25T01:26:29.629434+0000","last_undegraded":"2024-04-25T01:26:29.629434+0000","last_fullsized":"2024-04-25T01:26:29.629434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:38.886023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.657161+0000","last_change":"2024-04-25T01:25:16.757351+0000","last_active":"2024-04-25T01:26:29.657161+0000","last_peered":"2024-04-25T01:26:29.657161+0000","last_clean":"2024-04-25T01:26:29.657161+0000","last_became_active":"2024-04-25T01:25:16.756980+0000","last_became_peered":"2024-04-25T01:25:16.756980+0000","last_unstale":"2024-04-25T01:26:29.657161+0000","last_undegraded":"2024-04-25T01:26:29.657161+0000","last_fullsized":"2024-04-25T01:26:29.657161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:18.755302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.662243+0000","last_change":"2024-04-25T01:25:16.740265+0000","last_active":"2024-04-25T01:26:29.662243+0000","last_peered":"2024-04-25T01:26:29.662243+0000","last_clean":"2024-04-25T01:26:29.662243+0000","last_became_active":"2024-04-25T01:25:16.739345+0000","last_became_peered":"2024-04-25T01:25:16.739345+0000","last_unstale":"2024-04-25T01:26:29.662243+0000","last_undegraded":"2024-04-25T01:26:29.662243+0000","last_fullsized":"2024-04-25T01:26:29.662243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:55.399650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.652116+0000","last_change":"2024-04-25T01:25:16.753159+0000","last_active":"2024-04-25T01:26:29.652116+0000","last_peered":"2024-04-25T01:26:29.652116+0000","last_clean":"2024-04-25T01:26:29.652116+0000","last_became_active":"2024-04-25T01:25:16.751863+0000","last_became_peered":"2024-04-25T01:25:16.751863+0000","last_unstale":"2024-04-25T01:26:29.652116+0000","last_undegraded":"2024-04-25T01:26:29.652116+0000","last_fullsized":"2024-04-25T01:26:29.652116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:54.218096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658976+0000","last_change":"2024-04-25T01:25:16.738727+0000","last_active":"2024-04-25T01:26:29.658976+0000","last_peered":"2024-04-25T01:26:29.658976+0000","last_clean":"2024-04-25T01:26:29.658976+0000","last_became_active":"2024-04-25T01:25:16.738390+0000","last_became_peered":"2024-04-25T01:25:16.738390+0000","last_unstale":"2024-04-25T01:26:29.658976+0000","last_undegraded":"2024-04-25T01:26:29.658976+0000","last_fullsized":"2024-04-25T01:26:29.658976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:03.968710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.657176+0000","last_change":"2024-04-25T01:25:16.756974+0000","last_active":"2024-04-25T01:26:29.657176+0000","last_peered":"2024-04-25T01:26:29.657176+0000","last_clean":"2024-04-25T01:26:29.657176+0000","last_became_active":"2024-04-25T01:25:16.756549+0000","last_became_peered":"2024-04-25T01:25:16.756549+0000","last_unstale":"2024-04-25T01:26:29.657176+0000","last_undegraded":"2024-04-25T01:26:29.657176+0000","last_fullsized":"2024-04-25T01:26:29.657176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:49.042132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.660437+0000","last_change":"2024-04-25T01:25:16.745728+0000","last_active":"2024-04-25T01:26:29.660437+0000","last_peered":"2024-04-25T01:26:29.660437+0000","last_clean":"2024-04-25T01:26:29.660437+0000","last_became_active":"2024-04-25T01:25:16.745091+0000","last_became_peered":"2024-04-25T01:25:16.745091+0000","last_unstale":"2024-04-25T01:26:29.660437+0000","last_undegraded":"2024-04-25T01:26:29.660437+0000","last_fullsized":"2024-04-25T01:26:29.660437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:01.622335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.651892+0000","last_change":"2024-04-25T01:25:16.755146+0000","last_active":"2024-04-25T01:26:29.651892+0000","last_peered":"2024-04-25T01:26:29.651892+0000","last_clean":"2024-04-25T01:26:29.651892+0000","last_became_active":"2024-04-25T01:25:16.752713+0000","last_became_peered":"2024-04-25T01:25:16.752713+0000","last_unstale":"2024-04-25T01:26:29.651892+0000","last_undegraded":"2024-04-25T01:26:29.651892+0000","last_fullsized":"2024-04-25T01:26:29.651892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:41:31.967903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.642337+0000","last_change":"2024-04-25T01:25:16.751162+0000","last_active":"2024-04-25T01:26:29.642337+0000","last_peered":"2024-04-25T01:26:29.642337+0000","last_clean":"2024-04-25T01:26:29.642337+0000","last_became_active":"2024-04-25T01:25:16.750772+0000","last_became_peered":"2024-04-25T01:25:16.750772+0000","last_unstale":"2024-04-25T01:26:29.642337+0000","last_undegraded":"2024-04-25T01:26:29.642337+0000","last_fullsized":"2024-04-25T01:26:29.642337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:39.533120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.652081+0000","last_change":"2024-04-25T01:25:16.754367+0000","last_active":"2024-04-25T01:26:29.652081+0000","last_peered":"2024-04-25T01:26:29.652081+0000","last_clean":"2024-04-25T01:26:29.652081+0000","last_became_active":"2024-04-25T01:25:16.750324+0000","last_became_peered":"2024-04-25T01:25:16.750324+0000","last_unstale":"2024-04-25T01:26:29.652081+0000","last_undegraded":"2024-04-25T01:26:29.652081+0000","last_fullsized":"2024-04-25T01:26:29.652081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:17.606187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.642673+0000","last_change":"2024-04-25T01:25:16.753111+0000","last_active":"2024-04-25T01:26:29.642673+0000","last_peered":"2024-04-25T01:26:29.642673+0000","last_clean":"2024-04-25T01:26:29.642673+0000","last_became_active":"2024-04-25T01:25:16.752749+0000","last_became_peered":"2024-04-25T01:25:16.752749+0000","last_unstale":"2024-04-25T01:26:29.642673+0000","last_undegraded":"2024-04-25T01:26:29.642673+0000","last_fullsized":"2024-04-25T01:26:29.642673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:07:28.653568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661925+0000","last_change":"2024-04-25T01:25:16.744958+0000","last_active":"2024-04-25T01:26:29.661925+0000","last_peered":"2024-04-25T01:26:29.661925+0000","last_clean":"2024-04-25T01:26:29.661925+0000","last_became_active":"2024-04-25T01:25:16.744606+0000","last_became_peered":"2024-04-25T01:25:16.744606+0000","last_unstale":"2024-04-25T01:26:29.661925+0000","last_undegraded":"2024-04-25T01:26:29.661925+0000","last_fullsized":"2024-04-25T01:26:29.661925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:07:58.920942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643973+0000","last_change":"2024-04-25T01:25:16.743462+0000","last_active":"2024-04-25T01:26:29.643973+0000","last_peered":"2024-04-25T01:26:29.643973+0000","last_clean":"2024-04-25T01:26:29.643973+0000","last_became_active":"2024-04-25T01:25:16.743055+0000","last_became_peered":"2024-04-25T01:25:16.743055+0000","last_unstale":"2024-04-25T01:26:29.643973+0000","last_undegraded":"2024-04-25T01:26:29.643973+0000","last_fullsized":"2024-04-25T01:26:29.643973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:23:03.660065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.641408+0000","last_change":"2024-04-25T01:25:16.751446+0000","last_active":"2024-04-25T01:26:29.641408+0000","last_peered":"2024-04-25T01:26:29.641408+0000","last_clean":"2024-04-25T01:26:29.641408+0000","last_became_active":"2024-04-25T01:25:16.751107+0000","last_became_peered":"2024-04-25T01:25:16.751107+0000","last_unstale":"2024-04-25T01:26:29.641408+0000","last_undegraded":"2024-04-25T01:26:29.641408+0000","last_fullsized":"2024-04-25T01:26:29.641408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:52.275662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654267+0000","last_change":"2024-04-25T01:25:16.748478+0000","last_active":"2024-04-25T01:26:29.654267+0000","last_peered":"2024-04-25T01:26:29.654267+0000","last_clean":"2024-04-25T01:26:29.654267+0000","last_became_active":"2024-04-25T01:25:16.748064+0000","last_became_peered":"2024-04-25T01:25:16.748064+0000","last_unstale":"2024-04-25T01:26:29.654267+0000","last_undegraded":"2024-04-25T01:26:29.654267+0000","last_fullsized":"2024-04-25T01:26:29.654267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:45.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181977+0000","last_change":"2024-04-25T01:25:15.987611+0000","last_active":"2024-04-25T01:26:01.181977+0000","last_peered":"2024-04-25T01:26:01.181977+0000","last_clean":"2024-04-25T01:26:01.181977+0000","last_became_active":"2024-04-25T01:25:15.987300+0000","last_became_peered":"2024-04-25T01:25:15.987300+0000","last_unstale":"2024-04-25T01:26:01.181977+0000","last_undegraded":"2024-04-25T01:26:01.181977+0000","last_fullsized":"2024-04-25T01:26:01.181977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:24.636607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.645469+0000","last_change":"2024-04-25T01:25:16.751076+0000","last_active":"2024-04-25T01:26:29.645469+0000","last_peered":"2024-04-25T01:26:29.645469+0000","last_clean":"2024-04-25T01:26:29.645469+0000","last_became_active":"2024-04-25T01:25:16.750219+0000","last_became_peered":"2024-04-25T01:25:16.750219+0000","last_unstale":"2024-04-25T01:26:29.645469+0000","last_undegraded":"2024-04-25T01:26:29.645469+0000","last_fullsized":"2024-04-25T01:26:29.645469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:11.956004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.182371+0000","last_change":"2024-04-25T01:25:15.977434+0000","last_active":"2024-04-25T01:26:01.182371+0000","last_peered":"2024-04-25T01:26:01.182371+0000","last_clean":"2024-04-25T01:26:01.182371+0000","last_became_active":"2024-04-25T01:25:15.977095+0000","last_became_peered":"2024-04-25T01:25:15.977095+0000","last_unstale":"2024-04-25T01:26:01.182371+0000","last_undegraded":"2024-04-25T01:26:01.182371+0000","last_fullsized":"2024-04-25T01:26:01.182371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:01.102962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658893+0000","last_change":"2024-04-25T01:25:16.740061+0000","last_active":"2024-04-25T01:26:29.658893+0000","last_peered":"2024-04-25T01:26:29.658893+0000","last_clean":"2024-04-25T01:26:29.658893+0000","last_became_active":"2024-04-25T01:25:16.739593+0000","last_became_peered":"2024-04-25T01:25:16.739593+0000","last_unstale":"2024-04-25T01:26:29.658893+0000","last_undegraded":"2024-04-25T01:26:29.658893+0000","last_fullsized":"2024-04-25T01:26:29.658893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:32.711066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.180580+0000","last_change":"2024-04-25T01:25:15.984150+0000","last_active":"2024-04-25T01:26:01.180580+0000","last_peered":"2024-04-25T01:26:01.180580+0000","last_clean":"2024-04-25T01:26:01.180580+0000","last_became_active":"2024-04-25T01:25:15.983405+0000","last_became_peered":"2024-04-25T01:25:15.983405+0000","last_unstale":"2024-04-25T01:26:01.180580+0000","last_undegraded":"2024-04-25T01:26:01.180580+0000","last_fullsized":"2024-04-25T01:26:01.180580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:11:38.511851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.659171+0000","last_change":"2024-04-25T01:25:16.751549+0000","last_active":"2024-04-25T01:26:29.659171+0000","last_peered":"2024-04-25T01:26:29.659171+0000","last_clean":"2024-04-25T01:26:29.659171+0000","last_became_active":"2024-04-25T01:25:16.751167+0000","last_became_peered":"2024-04-25T01:25:16.751167+0000","last_unstale":"2024-04-25T01:26:29.659171+0000","last_undegraded":"2024-04-25T01:26:29.659171+0000","last_fullsized":"2024-04-25T01:26:29.659171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:21.260019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.179808+0000","last_change":"2024-04-25T01:25:15.977916+0000","last_active":"2024-04-25T01:26:01.179808+0000","last_peered":"2024-04-25T01:26:01.179808+0000","last_clean":"2024-04-25T01:26:01.179808+0000","last_became_active":"2024-04-25T01:25:15.977602+0000","last_became_peered":"2024-04-25T01:25:15.977602+0000","last_unstale":"2024-04-25T01:26:01.179808+0000","last_undegraded":"2024-04-25T01:26:01.179808+0000","last_fullsized":"2024-04-25T01:26:01.179808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:27.926489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.655417+0000","last_change":"2024-04-25T01:25:16.756728+0000","last_active":"2024-04-25T01:26:29.655417+0000","last_peered":"2024-04-25T01:26:29.655417+0000","last_clean":"2024-04-25T01:26:29.655417+0000","last_became_active":"2024-04-25T01:25:16.756363+0000","last_became_peered":"2024-04-25T01:25:16.756363+0000","last_unstale":"2024-04-25T01:26:29.655417+0000","last_undegraded":"2024-04-25T01:26:29.655417+0000","last_fullsized":"2024-04-25T01:26:29.655417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:06.582831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":104,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:37.192493+0000","last_change":"2024-04-25T01:25:15.987099+0000","last_active":"2024-04-25T01:26:37.192493+0000","last_peered":"2024-04-25T01:26:37.192493+0000","last_clean":"2024-04-25T01:26:37.192493+0000","last_became_active":"2024-04-25T01:25:15.986745+0000","last_became_peered":"2024-04-25T01:25:15.986745+0000","last_unstale":"2024-04-25T01:26:37.192493+0000","last_undegraded":"2024-04-25T01:26:37.192493+0000","last_fullsized":"2024-04-25T01:26:37.192493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:14.944342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.633696+0000","last_change":"2024-04-25T01:25:16.742735+0000","last_active":"2024-04-25T01:26:29.633696+0000","last_peered":"2024-04-25T01:26:29.633696+0000","last_clean":"2024-04-25T01:26:29.633696+0000","last_became_active":"2024-04-25T01:25:16.742391+0000","last_became_peered":"2024-04-25T01:25:16.742391+0000","last_unstale":"2024-04-25T01:26:29.633696+0000","last_undegraded":"2024-04-25T01:26:29.633696+0000","last_fullsized":"2024-04-25T01:26:29.633696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:12:05.783111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181320+0000","last_change":"2024-04-25T01:25:15.982605+0000","last_active":"2024-04-25T01:26:01.181320+0000","last_peered":"2024-04-25T01:26:01.181320+0000","last_clean":"2024-04-25T01:26:01.181320+0000","last_became_active":"2024-04-25T01:25:15.982264+0000","last_became_peered":"2024-04-25T01:25:15.982264+0000","last_unstale":"2024-04-25T01:26:01.181320+0000","last_undegraded":"2024-04-25T01:26:01.181320+0000","last_fullsized":"2024-04-25T01:26:01.181320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:09.960827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654239+0000","last_change":"2024-04-25T01:25:16.747784+0000","last_active":"2024-04-25T01:26:29.654239+0000","last_peered":"2024-04-25T01:26:29.654239+0000","last_clean":"2024-04-25T01:26:29.654239+0000","last_became_active":"2024-04-25T01:25:16.747432+0000","last_became_peered":"2024-04-25T01:25:16.747432+0000","last_unstale":"2024-04-25T01:26:29.654239+0000","last_undegraded":"2024-04-25T01:26:29.654239+0000","last_fullsized":"2024-04-25T01:26:29.654239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:40.602582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181269+0000","last_change":"2024-04-25T01:25:15.977370+0000","last_active":"2024-04-25T01:26:01.181269+0000","last_peered":"2024-04-25T01:26:01.181269+0000","last_clean":"2024-04-25T01:26:01.181269+0000","last_became_active":"2024-04-25T01:25:15.977010+0000","last_became_peered":"2024-04-25T01:25:15.977010+0000","last_unstale":"2024-04-25T01:26:01.181269+0000","last_undegraded":"2024-04-25T01:26:01.181269+0000","last_fullsized":"2024-04-25T01:26:01.181269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:41.239154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643866+0000","last_change":"2024-04-25T01:25:16.741490+0000","last_active":"2024-04-25T01:26:29.643866+0000","last_peered":"2024-04-25T01:26:29.643866+0000","last_clean":"2024-04-25T01:26:29.643866+0000","last_became_active":"2024-04-25T01:25:16.741148+0000","last_became_peered":"2024-04-25T01:25:16.741148+0000","last_unstale":"2024-04-25T01:26:29.643866+0000","last_undegraded":"2024-04-25T01:26:29.643866+0000","last_fullsized":"2024-04-25T01:26:29.643866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:50.918575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.192446+0000","last_change":"2024-04-25T01:25:15.988283+0000","last_active":"2024-04-25T01:26:01.192446+0000","last_peered":"2024-04-25T01:26:01.192446+0000","last_clean":"2024-04-25T01:26:01.192446+0000","last_became_active":"2024-04-25T01:25:15.987894+0000","last_became_peered":"2024-04-25T01:25:15.987894+0000","last_unstale":"2024-04-25T01:26:01.192446+0000","last_undegraded":"2024-04-25T01:26:01.192446+0000","last_fullsized":"2024-04-25T01:26:01.192446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:01.220170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661223+0000","last_change":"2024-04-25T01:26:01.287127+0000","last_active":"2024-04-25T01:26:29.661223+0000","last_peered":"2024-04-25T01:26:29.661223+0000","last_clean":"2024-04-25T01:26:29.661223+0000","last_became_active":"2024-04-25T01:26:01.286830+0000","last_became_peered":"2024-04-25T01:26:01.286830+0000","last_unstale":"2024-04-25T01:26:29.661223+0000","last_undegraded":"2024-04-25T01:26:29.661223+0000","last_fullsized":"2024-04-25T01:26:29.661223+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:42:33.695703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:33.186120+0000","last_change":"2024-04-25T01:25:15.981590+0000","last_active":"2024-04-25T01:26:33.186120+0000","last_peered":"2024-04-25T01:26:33.186120+0000","last_clean":"2024-04-25T01:26:33.186120+0000","last_became_active":"2024-04-25T01:25:15.981192+0000","last_became_peered":"2024-04-25T01:25:15.981192+0000","last_unstale":"2024-04-25T01:26:33.186120+0000","last_undegraded":"2024-04-25T01:26:33.186120+0000","last_fullsized":"2024-04-25T01:26:33.186120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:06.220365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.658412+0000","last_change":"2024-04-25T01:25:16.751414+0000","last_active":"2024-04-25T01:26:29.658412+0000","last_peered":"2024-04-25T01:26:29.658412+0000","last_clean":"2024-04-25T01:26:29.658412+0000","last_became_active":"2024-04-25T01:25:16.751003+0000","last_became_peered":"2024-04-25T01:25:16.751003+0000","last_unstale":"2024-04-25T01:26:29.658412+0000","last_undegraded":"2024-04-25T01:26:29.658412+0000","last_fullsized":"2024-04-25T01:26:29.658412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:49:03.146685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.286169+0000","last_change":"2024-04-25T01:25:15.974041+0000","last_active":"2024-04-25T01:26:01.286169+0000","last_peered":"2024-04-25T01:26:01.286169+0000","last_clean":"2024-04-25T01:26:01.286169+0000","last_became_active":"2024-04-25T01:25:15.973621+0000","last_became_peered":"2024-04-25T01:25:15.973621+0000","last_unstale":"2024-04-25T01:26:01.286169+0000","last_undegraded":"2024-04-25T01:26:01.286169+0000","last_fullsized":"2024-04-25T01:26:01.286169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:12.039161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.648005+0000","last_change":"2024-04-25T01:25:16.745295+0000","last_active":"2024-04-25T01:26:29.648005+0000","last_peered":"2024-04-25T01:26:29.648005+0000","last_clean":"2024-04-25T01:26:29.648005+0000","last_became_active":"2024-04-25T01:25:16.744931+0000","last_became_peered":"2024-04-25T01:25:16.744931+0000","last_unstale":"2024-04-25T01:26:29.648005+0000","last_undegraded":"2024-04-25T01:26:29.648005+0000","last_fullsized":"2024-04-25T01:26:29.648005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:48.993537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.193101+0000","last_change":"2024-04-25T01:25:15.980900+0000","last_active":"2024-04-25T01:26:01.193101+0000","last_peered":"2024-04-25T01:26:01.193101+0000","last_clean":"2024-04-25T01:26:01.193101+0000","last_became_active":"2024-04-25T01:25:15.980528+0000","last_became_peered":"2024-04-25T01:25:15.980528+0000","last_unstale":"2024-04-25T01:26:01.193101+0000","last_undegraded":"2024-04-25T01:26:01.193101+0000","last_fullsized":"2024-04-25T01:26:01.193101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:47.743151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.660628+0000","last_change":"2024-04-25T01:25:16.753886+0000","last_active":"2024-04-25T01:26:29.660628+0000","last_peered":"2024-04-25T01:26:29.660628+0000","last_clean":"2024-04-25T01:26:29.660628+0000","last_became_active":"2024-04-25T01:25:16.753522+0000","last_became_peered":"2024-04-25T01:25:16.753522+0000","last_unstale":"2024-04-25T01:26:29.660628+0000","last_undegraded":"2024-04-25T01:26:29.660628+0000","last_fullsized":"2024-04-25T01:26:29.660628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:12:46.103249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":225,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:34.238907+0000","last_change":"2024-04-25T01:26:01.194086+0000","last_active":"2024-04-25T01:26:34.238907+0000","last_peered":"2024-04-25T01:26:34.238907+0000","last_clean":"2024-04-25T01:26:34.238907+0000","last_became_active":"2024-04-25T01:26:01.186240+0000","last_became_peered":"2024-04-25T01:26:01.186240+0000","last_unstale":"2024-04-25T01:26:34.238907+0000","last_undegraded":"2024-04-25T01:26:34.238907+0000","last_fullsized":"2024-04-25T01:26:34.238907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:31.758268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":147,"num_read_kb":133,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643176+0000","last_change":"2024-04-25T01:25:16.739726+0000","last_active":"2024-04-25T01:26:29.643176+0000","last_peered":"2024-04-25T01:26:29.643176+0000","last_clean":"2024-04-25T01:26:29.643176+0000","last_became_active":"2024-04-25T01:25:16.739399+0000","last_became_peered":"2024-04-25T01:25:16.739399+0000","last_unstale":"2024-04-25T01:26:29.643176+0000","last_undegraded":"2024-04-25T01:26:29.643176+0000","last_fullsized":"2024-04-25T01:26:29.643176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:21:42.747061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.184400+0000","last_change":"2024-04-25T01:25:15.976731+0000","last_active":"2024-04-25T01:26:01.184400+0000","last_peered":"2024-04-25T01:26:01.184400+0000","last_clean":"2024-04-25T01:26:01.184400+0000","last_became_active":"2024-04-25T01:25:15.976405+0000","last_became_peered":"2024-04-25T01:25:15.976405+0000","last_unstale":"2024-04-25T01:26:01.184400+0000","last_undegraded":"2024-04-25T01:26:01.184400+0000","last_fullsized":"2024-04-25T01:26:01.184400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:21.612389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.659889+0000","last_change":"2024-04-25T01:25:16.741378+0000","last_active":"2024-04-25T01:26:29.659889+0000","last_peered":"2024-04-25T01:26:29.659889+0000","last_clean":"2024-04-25T01:26:29.659889+0000","last_became_active":"2024-04-25T01:25:16.741066+0000","last_became_peered":"2024-04-25T01:25:16.741066+0000","last_unstale":"2024-04-25T01:26:29.659889+0000","last_undegraded":"2024-04-25T01:26:29.659889+0000","last_fullsized":"2024-04-25T01:26:29.659889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:17:37.288181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.192510+0000","last_change":"2024-04-25T01:25:15.988288+0000","last_active":"2024-04-25T01:26:01.192510+0000","last_peered":"2024-04-25T01:26:01.192510+0000","last_clean":"2024-04-25T01:26:01.192510+0000","last_became_active":"2024-04-25T01:25:15.987860+0000","last_became_peered":"2024-04-25T01:25:15.987860+0000","last_unstale":"2024-04-25T01:26:01.192510+0000","last_undegraded":"2024-04-25T01:26:01.192510+0000","last_fullsized":"2024-04-25T01:26:01.192510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:23.634199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.663569+0000","last_change":"2024-04-25T01:25:16.743784+0000","last_active":"2024-04-25T01:26:29.663569+0000","last_peered":"2024-04-25T01:26:29.663569+0000","last_clean":"2024-04-25T01:26:29.663569+0000","last_became_active":"2024-04-25T01:25:16.743435+0000","last_became_peered":"2024-04-25T01:25:16.743435+0000","last_unstale":"2024-04-25T01:26:29.663569+0000","last_undegraded":"2024-04-25T01:26:29.663569+0000","last_fullsized":"2024-04-25T01:26:29.663569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:52:08.297617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":102,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:33.186829+0000","last_change":"2024-04-25T01:25:15.978201+0000","last_active":"2024-04-25T01:26:33.186829+0000","last_peered":"2024-04-25T01:26:33.186829+0000","last_clean":"2024-04-25T01:26:33.186829+0000","last_became_active":"2024-04-25T01:25:15.977832+0000","last_became_peered":"2024-04-25T01:25:15.977832+0000","last_unstale":"2024-04-25T01:26:33.186829+0000","last_undegraded":"2024-04-25T01:26:33.186829+0000","last_fullsized":"2024-04-25T01:26:33.186829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:05:42.483969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661981+0000","last_change":"2024-04-25T01:25:16.750944+0000","last_active":"2024-04-25T01:26:29.661981+0000","last_peered":"2024-04-25T01:26:29.661981+0000","last_clean":"2024-04-25T01:26:29.661981+0000","last_became_active":"2024-04-25T01:25:16.750588+0000","last_became_peered":"2024-04-25T01:25:16.750588+0000","last_unstale":"2024-04-25T01:26:29.661981+0000","last_undegraded":"2024-04-25T01:26:29.661981+0000","last_fullsized":"2024-04-25T01:26:29.661981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:31.473184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.202567+0000","last_change":"2024-04-25T01:26:01.202137+0000","last_active":"2024-04-25T01:26:01.202567+0000","last_peered":"2024-04-25T01:26:01.202567+0000","last_clean":"2024-04-25T01:26:01.202567+0000","last_became_active":"2024-04-25T01:26:01.186142+0000","last_became_peered":"2024-04-25T01:26:01.186142+0000","last_unstale":"2024-04-25T01:26:01.202567+0000","last_undegraded":"2024-04-25T01:26:01.202567+0000","last_fullsized":"2024-04-25T01:26:01.202567+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:22:36.628455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.653831+0000","last_change":"2024-04-25T01:25:16.748178+0000","last_active":"2024-04-25T01:26:29.653831+0000","last_peered":"2024-04-25T01:26:29.653831+0000","last_clean":"2024-04-25T01:26:29.653831+0000","last_became_active":"2024-04-25T01:25:16.747868+0000","last_became_peered":"2024-04-25T01:25:16.747868+0000","last_unstale":"2024-04-25T01:26:29.653831+0000","last_undegraded":"2024-04-25T01:26:29.653831+0000","last_fullsized":"2024-04-25T01:26:29.653831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:06.378387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.179808+0000","last_change":"2024-04-25T01:25:15.981651+0000","last_active":"2024-04-25T01:26:01.179808+0000","last_peered":"2024-04-25T01:26:01.179808+0000","last_clean":"2024-04-25T01:26:01.179808+0000","last_became_active":"2024-04-25T01:25:15.981245+0000","last_became_peered":"2024-04-25T01:25:15.981245+0000","last_unstale":"2024-04-25T01:26:01.179808+0000","last_undegraded":"2024-04-25T01:26:01.179808+0000","last_fullsized":"2024-04-25T01:26:01.179808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:33:20.009347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.628024+0000","last_change":"2024-04-25T01:25:16.756873+0000","last_active":"2024-04-25T01:26:29.628024+0000","last_peered":"2024-04-25T01:26:29.628024+0000","last_clean":"2024-04-25T01:26:29.628024+0000","last_became_active":"2024-04-25T01:25:16.756505+0000","last_became_peered":"2024-04-25T01:25:16.756505+0000","last_unstale":"2024-04-25T01:26:29.628024+0000","last_undegraded":"2024-04-25T01:26:29.628024+0000","last_fullsized":"2024-04-25T01:26:29.628024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:23.412303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181032+0000","last_change":"2024-04-25T01:25:15.983640+0000","last_active":"2024-04-25T01:26:01.181032+0000","last_peered":"2024-04-25T01:26:01.181032+0000","last_clean":"2024-04-25T01:26:01.181032+0000","last_became_active":"2024-04-25T01:25:15.983280+0000","last_became_peered":"2024-04-25T01:25:15.983280+0000","last_unstale":"2024-04-25T01:26:01.181032+0000","last_undegraded":"2024-04-25T01:26:01.181032+0000","last_fullsized":"2024-04-25T01:26:01.181032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:00:18.493139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643060+0000","last_change":"2024-04-25T01:25:16.751616+0000","last_active":"2024-04-25T01:26:29.643060+0000","last_peered":"2024-04-25T01:26:29.643060+0000","last_clean":"2024-04-25T01:26:29.643060+0000","last_became_active":"2024-04-25T01:25:16.751254+0000","last_became_peered":"2024-04-25T01:25:16.751254+0000","last_unstale":"2024-04-25T01:26:29.643060+0000","last_undegraded":"2024-04-25T01:26:29.643060+0000","last_fullsized":"2024-04-25T01:26:29.643060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:55.556054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.286025+0000","last_change":"2024-04-25T01:25:15.977774+0000","last_active":"2024-04-25T01:26:01.286025+0000","last_peered":"2024-04-25T01:26:01.286025+0000","last_clean":"2024-04-25T01:26:01.286025+0000","last_became_active":"2024-04-25T01:25:15.977398+0000","last_became_peered":"2024-04-25T01:25:15.977398+0000","last_unstale":"2024-04-25T01:26:01.286025+0000","last_undegraded":"2024-04-25T01:26:01.286025+0000","last_fullsized":"2024-04-25T01:26:01.286025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:33.729091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.645162+0000","last_change":"2024-04-25T01:26:01.185468+0000","last_active":"2024-04-25T01:26:29.645162+0000","last_peered":"2024-04-25T01:26:29.645162+0000","last_clean":"2024-04-25T01:26:29.645162+0000","last_became_active":"2024-04-25T01:26:01.184888+0000","last_became_peered":"2024-04-25T01:26:01.184888+0000","last_unstale":"2024-04-25T01:26:29.645162+0000","last_undegraded":"2024-04-25T01:26:29.645162+0000","last_fullsized":"2024-04-25T01:26:29.645162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:59.565972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.653980+0000","last_change":"2024-04-25T01:25:16.746447+0000","last_active":"2024-04-25T01:26:29.653980+0000","last_peered":"2024-04-25T01:26:29.653980+0000","last_clean":"2024-04-25T01:26:29.653980+0000","last_became_active":"2024-04-25T01:25:16.745978+0000","last_became_peered":"2024-04-25T01:25:16.745978+0000","last_unstale":"2024-04-25T01:26:29.653980+0000","last_undegraded":"2024-04-25T01:26:29.653980+0000","last_fullsized":"2024-04-25T01:26:29.653980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:41.249662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:35.188962+0000","last_change":"2024-04-25T01:25:15.987083+0000","last_active":"2024-04-25T01:26:35.188962+0000","last_peered":"2024-04-25T01:26:35.188962+0000","last_clean":"2024-04-25T01:26:35.188962+0000","last_became_active":"2024-04-25T01:25:15.986632+0000","last_became_peered":"2024-04-25T01:25:15.986632+0000","last_unstale":"2024-04-25T01:26:35.188962+0000","last_undegraded":"2024-04-25T01:26:35.188962+0000","last_fullsized":"2024-04-25T01:26:35.188962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:06:44.792239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.637980+0000","last_change":"2024-04-25T01:25:16.742007+0000","last_active":"2024-04-25T01:26:29.637980+0000","last_peered":"2024-04-25T01:26:29.637980+0000","last_clean":"2024-04-25T01:26:29.637980+0000","last_became_active":"2024-04-25T01:25:16.741464+0000","last_became_peered":"2024-04-25T01:25:16.741464+0000","last_unstale":"2024-04-25T01:26:29.637980+0000","last_undegraded":"2024-04-25T01:26:29.637980+0000","last_fullsized":"2024-04-25T01:26:29.637980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:32.171648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181599+0000","last_change":"2024-04-25T01:25:15.978255+0000","last_active":"2024-04-25T01:26:01.181599+0000","last_peered":"2024-04-25T01:26:01.181599+0000","last_clean":"2024-04-25T01:26:01.181599+0000","last_became_active":"2024-04-25T01:25:15.977860+0000","last_became_peered":"2024-04-25T01:25:15.977860+0000","last_unstale":"2024-04-25T01:26:01.181599+0000","last_undegraded":"2024-04-25T01:26:01.181599+0000","last_fullsized":"2024-04-25T01:26:01.181599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:35.516137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.285521+0000","last_change":"2024-04-25T01:23:58.204099+0000","last_active":"2024-04-25T01:26:01.285521+0000","last_peered":"2024-04-25T01:26:01.285521+0000","last_clean":"2024-04-25T01:26:01.285521+0000","last_became_active":"2024-04-25T01:23:58.203533+0000","last_became_peered":"2024-04-25T01:23:58.203533+0000","last_unstale":"2024-04-25T01:26:01.285521+0000","last_undegraded":"2024-04-25T01:26:01.285521+0000","last_fullsized":"2024-04-25T01:26:01.285521+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:20:28.584758+0000","last_clean_scrub_stamp":"2024-04-25T01:20:28.584758+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:17.504360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.653755+0000","last_change":"2024-04-25T01:26:01.186051+0000","last_active":"2024-04-25T01:26:29.653755+0000","last_peered":"2024-04-25T01:26:29.653755+0000","last_clean":"2024-04-25T01:26:29.653755+0000","last_became_active":"2024-04-25T01:26:01.185684+0000","last_became_peered":"2024-04-25T01:26:01.185684+0000","last_unstale":"2024-04-25T01:26:29.653755+0000","last_undegraded":"2024-04-25T01:26:29.653755+0000","last_fullsized":"2024-04-25T01:26:29.653755+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:46.754769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.285610+0000","last_change":"2024-04-25T01:25:15.986400+0000","last_active":"2024-04-25T01:26:01.285610+0000","last_peered":"2024-04-25T01:26:01.285610+0000","last_clean":"2024-04-25T01:26:01.285610+0000","last_became_active":"2024-04-25T01:25:15.985975+0000","last_became_peered":"2024-04-25T01:25:15.985975+0000","last_unstale":"2024-04-25T01:26:01.285610+0000","last_undegraded":"2024-04-25T01:26:01.285610+0000","last_fullsized":"2024-04-25T01:26:01.285610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:12.666394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.180431+0000","last_change":"2024-04-25T01:25:15.981370+0000","last_active":"2024-04-25T01:26:01.180431+0000","last_peered":"2024-04-25T01:26:01.180431+0000","last_clean":"2024-04-25T01:26:01.180431+0000","last_became_active":"2024-04-25T01:25:15.980995+0000","last_became_peered":"2024-04-25T01:25:15.980995+0000","last_unstale":"2024-04-25T01:26:01.180431+0000","last_undegraded":"2024-04-25T01:26:01.180431+0000","last_fullsized":"2024-04-25T01:26:01.180431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:29.754441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654232+0000","last_change":"2024-04-25T01:25:16.744887+0000","last_active":"2024-04-25T01:26:29.654232+0000","last_peered":"2024-04-25T01:26:29.654232+0000","last_clean":"2024-04-25T01:26:29.654232+0000","last_became_active":"2024-04-25T01:25:16.744538+0000","last_became_peered":"2024-04-25T01:25:16.744538+0000","last_unstale":"2024-04-25T01:26:29.654232+0000","last_undegraded":"2024-04-25T01:26:29.654232+0000","last_fullsized":"2024-04-25T01:26:29.654232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:32.132521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.180875+0000","last_change":"2024-04-25T01:25:15.990181+0000","last_active":"2024-04-25T01:26:01.180875+0000","last_peered":"2024-04-25T01:26:01.180875+0000","last_clean":"2024-04-25T01:26:01.180875+0000","last_became_active":"2024-04-25T01:25:15.989816+0000","last_became_peered":"2024-04-25T01:25:15.989816+0000","last_unstale":"2024-04-25T01:26:01.180875+0000","last_undegraded":"2024-04-25T01:26:01.180875+0000","last_fullsized":"2024-04-25T01:26:01.180875+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:17:58.377057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.651935+0000","last_change":"2024-04-25T01:25:16.746199+0000","last_active":"2024-04-25T01:26:29.651935+0000","last_peered":"2024-04-25T01:26:29.651935+0000","last_clean":"2024-04-25T01:26:29.651935+0000","last_became_active":"2024-04-25T01:25:16.745857+0000","last_became_peered":"2024-04-25T01:25:16.745857+0000","last_unstale":"2024-04-25T01:26:29.651935+0000","last_undegraded":"2024-04-25T01:26:29.651935+0000","last_fullsized":"2024-04-25T01:26:29.651935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:56:21.392309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.182299+0000","last_change":"2024-04-25T01:25:15.981173+0000","last_active":"2024-04-25T01:26:01.182299+0000","last_peered":"2024-04-25T01:26:01.182299+0000","last_clean":"2024-04-25T01:26:01.182299+0000","last_became_active":"2024-04-25T01:25:15.980718+0000","last_became_peered":"2024-04-25T01:25:15.980718+0000","last_unstale":"2024-04-25T01:26:01.182299+0000","last_undegraded":"2024-04-25T01:26:01.182299+0000","last_fullsized":"2024-04-25T01:26:01.182299+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:55:54.086543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.649554+0000","last_change":"2024-04-25T01:25:16.753987+0000","last_active":"2024-04-25T01:26:29.649554+0000","last_peered":"2024-04-25T01:26:29.649554+0000","last_clean":"2024-04-25T01:26:29.649554+0000","last_became_active":"2024-04-25T01:25:16.753546+0000","last_became_peered":"2024-04-25T01:25:16.753546+0000","last_unstale":"2024-04-25T01:26:29.649554+0000","last_undegraded":"2024-04-25T01:26:29.649554+0000","last_fullsized":"2024-04-25T01:26:29.649554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:10.318501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.184414+0000","last_change":"2024-04-25T01:25:15.980938+0000","last_active":"2024-04-25T01:26:01.184414+0000","last_peered":"2024-04-25T01:26:01.184414+0000","last_clean":"2024-04-25T01:26:01.184414+0000","last_became_active":"2024-04-25T01:25:15.980586+0000","last_became_peered":"2024-04-25T01:25:15.980586+0000","last_unstale":"2024-04-25T01:26:01.184414+0000","last_undegraded":"2024-04-25T01:26:01.184414+0000","last_fullsized":"2024-04-25T01:26:01.184414+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:00.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.605599+0000","last_change":"2024-04-25T01:25:16.749711+0000","last_active":"2024-04-25T01:26:29.605599+0000","last_peered":"2024-04-25T01:26:29.605599+0000","last_clean":"2024-04-25T01:26:29.605599+0000","last_became_active":"2024-04-25T01:25:16.749218+0000","last_became_peered":"2024-04-25T01:25:16.749218+0000","last_unstale":"2024-04-25T01:26:29.605599+0000","last_undegraded":"2024-04-25T01:26:29.605599+0000","last_fullsized":"2024-04-25T01:26:29.605599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:56.555204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181476+0000","last_change":"2024-04-25T01:25:15.988257+0000","last_active":"2024-04-25T01:26:01.181476+0000","last_peered":"2024-04-25T01:26:01.181476+0000","last_clean":"2024-04-25T01:26:01.181476+0000","last_became_active":"2024-04-25T01:25:15.987469+0000","last_became_peered":"2024-04-25T01:25:15.987469+0000","last_unstale":"2024-04-25T01:26:01.181476+0000","last_undegraded":"2024-04-25T01:26:01.181476+0000","last_fullsized":"2024-04-25T01:26:01.181476+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:54:29.849868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.648521+0000","last_change":"2024-04-25T01:25:16.746686+0000","last_active":"2024-04-25T01:26:29.648521+0000","last_peered":"2024-04-25T01:26:29.648521+0000","last_clean":"2024-04-25T01:26:29.648521+0000","last_became_active":"2024-04-25T01:25:16.746028+0000","last_became_peered":"2024-04-25T01:25:16.746028+0000","last_unstale":"2024-04-25T01:26:29.648521+0000","last_undegraded":"2024-04-25T01:26:29.648521+0000","last_fullsized":"2024-04-25T01:26:29.648521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:34:35.172207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.189978+0000","last_change":"2024-04-25T01:26:01.189508+0000","last_active":"2024-04-25T01:26:01.189978+0000","last_peered":"2024-04-25T01:26:01.189978+0000","last_clean":"2024-04-25T01:26:01.189978+0000","last_became_active":"2024-04-25T01:26:01.184031+0000","last_became_peered":"2024-04-25T01:26:01.184031+0000","last_unstale":"2024-04-25T01:26:01.189978+0000","last_undegraded":"2024-04-25T01:26:01.189978+0000","last_fullsized":"2024-04-25T01:26:01.189978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:50.354208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.651185+0000","last_change":"2024-04-25T01:25:16.752510+0000","last_active":"2024-04-25T01:26:29.651185+0000","last_peered":"2024-04-25T01:26:29.651185+0000","last_clean":"2024-04-25T01:26:29.651185+0000","last_became_active":"2024-04-25T01:25:16.751889+0000","last_became_peered":"2024-04-25T01:25:16.751889+0000","last_unstale":"2024-04-25T01:26:29.651185+0000","last_undegraded":"2024-04-25T01:26:29.651185+0000","last_fullsized":"2024-04-25T01:26:29.651185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:44.628980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.179564+0000","last_change":"2024-04-25T01:25:15.983638+0000","last_active":"2024-04-25T01:26:01.179564+0000","last_peered":"2024-04-25T01:26:01.179564+0000","last_clean":"2024-04-25T01:26:01.179564+0000","last_became_active":"2024-04-25T01:25:15.983293+0000","last_became_peered":"2024-04-25T01:25:15.983293+0000","last_unstale":"2024-04-25T01:26:01.179564+0000","last_undegraded":"2024-04-25T01:26:01.179564+0000","last_fullsized":"2024-04-25T01:26:01.179564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:04:27.495446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.659977+0000","last_change":"2024-04-25T01:25:16.736420+0000","last_active":"2024-04-25T01:26:29.659977+0000","last_peered":"2024-04-25T01:26:29.659977+0000","last_clean":"2024-04-25T01:26:29.659977+0000","last_became_active":"2024-04-25T01:25:16.736031+0000","last_became_peered":"2024-04-25T01:25:16.736031+0000","last_unstale":"2024-04-25T01:26:29.659977+0000","last_undegraded":"2024-04-25T01:26:29.659977+0000","last_fullsized":"2024-04-25T01:26:29.659977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:33.380905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.185349+0000","last_change":"2024-04-25T01:25:15.982201+0000","last_active":"2024-04-25T01:26:01.185349+0000","last_peered":"2024-04-25T01:26:01.185349+0000","last_clean":"2024-04-25T01:26:01.185349+0000","last_became_active":"2024-04-25T01:25:15.981820+0000","last_became_peered":"2024-04-25T01:25:15.981820+0000","last_unstale":"2024-04-25T01:26:01.185349+0000","last_undegraded":"2024-04-25T01:26:01.185349+0000","last_fullsized":"2024-04-25T01:26:01.185349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:12.149417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.660402+0000","last_change":"2024-04-25T01:25:16.739126+0000","last_active":"2024-04-25T01:26:29.660402+0000","last_peered":"2024-04-25T01:26:29.660402+0000","last_clean":"2024-04-25T01:26:29.660402+0000","last_became_active":"2024-04-25T01:25:16.738762+0000","last_became_peered":"2024-04-25T01:25:16.738762+0000","last_unstale":"2024-04-25T01:26:29.660402+0000","last_undegraded":"2024-04-25T01:26:29.660402+0000","last_fullsized":"2024-04-25T01:26:29.660402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:00:47.661399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.192997+0000","last_change":"2024-04-25T01:25:15.985732+0000","last_active":"2024-04-25T01:26:01.192997+0000","last_peered":"2024-04-25T01:26:01.192997+0000","last_clean":"2024-04-25T01:26:01.192997+0000","last_became_active":"2024-04-25T01:25:15.985300+0000","last_became_peered":"2024-04-25T01:25:15.985300+0000","last_unstale":"2024-04-25T01:26:01.192997+0000","last_undegraded":"2024-04-25T01:26:01.192997+0000","last_fullsized":"2024-04-25T01:26:01.192997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:27:01.180165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.657443+0000","last_change":"2024-04-25T01:25:16.746639+0000","last_active":"2024-04-25T01:26:29.657443+0000","last_peered":"2024-04-25T01:26:29.657443+0000","last_clean":"2024-04-25T01:26:29.657443+0000","last_became_active":"2024-04-25T01:25:16.746291+0000","last_became_peered":"2024-04-25T01:25:16.746291+0000","last_unstale":"2024-04-25T01:26:29.657443+0000","last_undegraded":"2024-04-25T01:26:29.657443+0000","last_fullsized":"2024-04-25T01:26:29.657443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:31.346243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:01.181626+0000","last_change":"2024-04-25T01:25:15.989193+0000","last_active":"2024-04-25T01:26:01.181626+0000","last_peered":"2024-04-25T01:26:01.181626+0000","last_clean":"2024-04-25T01:26:01.181626+0000","last_became_active":"2024-04-25T01:25:15.988811+0000","last_became_peered":"2024-04-25T01:25:15.988811+0000","last_unstale":"2024-04-25T01:26:01.181626+0000","last_undegraded":"2024-04-25T01:26:01.181626+0000","last_fullsized":"2024-04-25T01:26:01.181626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:14.958400+0000","last_clean_scrub_stamp":"2024-04-25T01:25:14.958400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:56.030398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.656944+0000","last_change":"2024-04-25T01:25:16.742209+0000","last_active":"2024-04-25T01:26:29.656944+0000","last_peered":"2024-04-25T01:26:29.656944+0000","last_clean":"2024-04-25T01:26:29.656944+0000","last_became_active":"2024-04-25T01:25:16.741810+0000","last_became_peered":"2024-04-25T01:25:16.741810+0000","last_unstale":"2024-04-25T01:26:29.656944+0000","last_undegraded":"2024-04-25T01:26:29.656944+0000","last_fullsized":"2024-04-25T01:26:29.656944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:38.008108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.648828+0000","last_change":"2024-04-25T01:25:16.747264+0000","last_active":"2024-04-25T01:26:29.648828+0000","last_peered":"2024-04-25T01:26:29.648828+0000","last_clean":"2024-04-25T01:26:29.648828+0000","last_became_active":"2024-04-25T01:25:16.746514+0000","last_became_peered":"2024-04-25T01:25:16.746514+0000","last_unstale":"2024-04-25T01:26:29.648828+0000","last_undegraded":"2024-04-25T01:26:29.648828+0000","last_fullsized":"2024-04-25T01:26:29.648828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:14.672062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.646761+0000","last_change":"2024-04-25T01:26:01.194587+0000","last_active":"2024-04-25T01:26:29.646761+0000","last_peered":"2024-04-25T01:26:29.646761+0000","last_clean":"2024-04-25T01:26:29.646761+0000","last_became_active":"2024-04-25T01:26:01.194035+0000","last_became_peered":"2024-04-25T01:26:01.194035+0000","last_unstale":"2024-04-25T01:26:29.646761+0000","last_undegraded":"2024-04-25T01:26:29.646761+0000","last_fullsized":"2024-04-25T01:26:29.646761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:29.304743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.644292+0000","last_change":"2024-04-25T01:25:16.752000+0000","last_active":"2024-04-25T01:26:29.644292+0000","last_peered":"2024-04-25T01:26:29.644292+0000","last_clean":"2024-04-25T01:26:29.644292+0000","last_became_active":"2024-04-25T01:25:16.749443+0000","last_became_peered":"2024-04-25T01:25:16.749443+0000","last_unstale":"2024-04-25T01:26:29.644292+0000","last_undegraded":"2024-04-25T01:26:29.644292+0000","last_fullsized":"2024-04-25T01:26:29.644292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:02.287719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.643640+0000","last_change":"2024-04-25T01:25:16.742510+0000","last_active":"2024-04-25T01:26:29.643640+0000","last_peered":"2024-04-25T01:26:29.643640+0000","last_clean":"2024-04-25T01:26:29.643640+0000","last_became_active":"2024-04-25T01:25:16.742162+0000","last_became_peered":"2024-04-25T01:25:16.742162+0000","last_unstale":"2024-04-25T01:26:29.643640+0000","last_undegraded":"2024-04-25T01:26:29.643640+0000","last_fullsized":"2024-04-25T01:26:29.643640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:02.924505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.654633+0000","last_change":"2024-04-25T01:25:16.756224+0000","last_active":"2024-04-25T01:26:29.654633+0000","last_peered":"2024-04-25T01:26:29.654633+0000","last_clean":"2024-04-25T01:26:29.654633+0000","last_became_active":"2024-04-25T01:25:16.755868+0000","last_became_peered":"2024-04-25T01:25:16.755868+0000","last_unstale":"2024-04-25T01:26:29.654633+0000","last_undegraded":"2024-04-25T01:26:29.654633+0000","last_fullsized":"2024-04-25T01:26:29.654633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:53.078309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.637047+0000","last_change":"2024-04-25T01:26:01.184911+0000","last_active":"2024-04-25T01:26:29.637047+0000","last_peered":"2024-04-25T01:26:29.637047+0000","last_clean":"2024-04-25T01:26:29.637047+0000","last_became_active":"2024-04-25T01:26:01.184299+0000","last_became_peered":"2024-04-25T01:26:01.184299+0000","last_unstale":"2024-04-25T01:26:29.637047+0000","last_undegraded":"2024-04-25T01:26:29.637047+0000","last_fullsized":"2024-04-25T01:26:29.637047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:32.156598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.649699+0000","last_change":"2024-04-25T01:25:16.745294+0000","last_active":"2024-04-25T01:26:29.649699+0000","last_peered":"2024-04-25T01:26:29.649699+0000","last_clean":"2024-04-25T01:26:29.649699+0000","last_became_active":"2024-04-25T01:25:16.743285+0000","last_became_peered":"2024-04-25T01:25:16.743285+0000","last_unstale":"2024-04-25T01:26:29.649699+0000","last_undegraded":"2024-04-25T01:26:29.649699+0000","last_fullsized":"2024-04-25T01:26:29.649699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:36.189339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.638875+0000","last_change":"2024-04-25T01:26:01.185479+0000","last_active":"2024-04-25T01:26:29.638875+0000","last_peered":"2024-04-25T01:26:29.638875+0000","last_clean":"2024-04-25T01:26:29.638875+0000","last_became_active":"2024-04-25T01:26:01.185073+0000","last_became_peered":"2024-04-25T01:26:01.185073+0000","last_unstale":"2024-04-25T01:26:29.638875+0000","last_undegraded":"2024-04-25T01:26:29.638875+0000","last_fullsized":"2024-04-25T01:26:29.638875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:32.184596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.635518+0000","last_change":"2024-04-25T01:26:01.184323+0000","last_active":"2024-04-25T01:26:29.635518+0000","last_peered":"2024-04-25T01:26:29.635518+0000","last_clean":"2024-04-25T01:26:29.635518+0000","last_became_active":"2024-04-25T01:26:01.183875+0000","last_became_peered":"2024-04-25T01:26:01.183875+0000","last_unstale":"2024-04-25T01:26:29.635518+0000","last_undegraded":"2024-04-25T01:26:29.635518+0000","last_fullsized":"2024-04-25T01:26:29.635518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:46:54.475788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.634802+0000","last_change":"2024-04-25T01:25:16.750026+0000","last_active":"2024-04-25T01:26:29.634802+0000","last_peered":"2024-04-25T01:26:29.634802+0000","last_clean":"2024-04-25T01:26:29.634802+0000","last_became_active":"2024-04-25T01:25:16.749588+0000","last_became_peered":"2024-04-25T01:25:16.749588+0000","last_unstale":"2024-04-25T01:26:29.634802+0000","last_undegraded":"2024-04-25T01:26:29.634802+0000","last_fullsized":"2024-04-25T01:26:29.634802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:09.357719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.662830+0000","last_change":"2024-04-25T01:25:16.743440+0000","last_active":"2024-04-25T01:26:29.662830+0000","last_peered":"2024-04-25T01:26:29.662830+0000","last_clean":"2024-04-25T01:26:29.662830+0000","last_became_active":"2024-04-25T01:25:16.742904+0000","last_became_peered":"2024-04-25T01:25:16.742904+0000","last_unstale":"2024-04-25T01:26:29.662830+0000","last_undegraded":"2024-04-25T01:26:29.662830+0000","last_fullsized":"2024-04-25T01:26:29.662830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:17.380260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.661691+0000","last_change":"2024-04-25T01:25:16.738305+0000","last_active":"2024-04-25T01:26:29.661691+0000","last_peered":"2024-04-25T01:26:29.661691+0000","last_clean":"2024-04-25T01:26:29.661691+0000","last_became_active":"2024-04-25T01:25:16.737907+0000","last_became_peered":"2024-04-25T01:25:16.737907+0000","last_unstale":"2024-04-25T01:26:29.661691+0000","last_undegraded":"2024-04-25T01:26:29.661691+0000","last_fullsized":"2024-04-25T01:26:29.661691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:29.501388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:26:29.633237+0000","last_change":"2024-04-25T01:25:16.757755+0000","last_active":"2024-04-25T01:26:29.633237+0000","last_peered":"2024-04-25T01:26:29.633237+0000","last_clean":"2024-04-25T01:26:29.633237+0000","last_became_active":"2024-04-25T01:25:16.757374+0000","last_became_peered":"2024-04-25T01:25:16.757374+0000","last_unstale":"2024-04-25T01:26:29.633237+0000","last_undegraded":"2024-04-25T01:26:29.633237+0000","last_fullsized":"2024-04-25T01:26:29.633237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:15.959891+0000","last_clean_scrub_stamp":"2024-04-25T01:25:15.959891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:10:46.848407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250979,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":468,"num_read_kb":302,"num_write":163,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":61158,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":196311,"data_compressed":13269,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":456499,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":1109695,"data_compressed":44224,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456623,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":456431,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1048614,"data_compressed":42368,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456012,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":517100,"data_compressed":7667,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":457283,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":457155,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":456323,"data_compressed":5811,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":518276,"data_compressed":15368,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1047766,"data_compressed":42368,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34917,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36557,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":62471,"data_compressed":9557,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63405,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:26:41.175 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T01:26:41.175 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:26:41.175 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:26:41.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T01:26:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:41.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:41 smithi016 ceph-mon[29153]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:26:41.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:41 smithi016 ceph-mon[29153]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:26:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:41 smithi038 ceph-mon[32841]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:26:41.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:41 smithi038 ceph-mon[32841]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:26:41.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:41 smithi156 ceph-mon[32631]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:26:41.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:41 smithi156 ceph-mon[32631]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:26:43.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 -- 172.21.15.16:0/1275804789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 msgr2=0x7efda810f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 --2- 172.21.15.16:0/1275804789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda810f0f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efda000b0a0 tx=0x7efda000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 -- 172.21.15.16:0/1275804789 shutdown_connections 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 --2- 172.21.15.16:0/1275804789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda810f0f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 --2- 172.21.15.16:0/1275804789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda81082b0 0x7efda8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 --2- 172.21.15.16:0/1275804789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 0x7efda806adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 -- 172.21.15.16:0/1275804789 >> 172.21.15.16:0/1275804789 conn(0x7efda80780e0 msgr2=0x7efda80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 -- 172.21.15.16:0/1275804789 shutdown_connections 2024-04-25T01:26:43.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.304+0000 7efdacce4700 1 -- 172.21.15.16:0/1275804789 wait complete. 2024-04-25T01:26:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 Processor -- start 2024-04-25T01:26:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 -- start start 2024-04-25T01:26:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 0x7efda81a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda81a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda81082b0 0x7efda819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:43.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efda81123b0 con 0x7efda806b300 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efda8112230 con 0x7efda81082b0 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efdacce4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efda8112530 con 0x7efda806a960 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda81a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda81a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57856/0 (socket says 172.21.15.16:57856) 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 -- 172.21.15.16:0/2301752163 learned_addr learned my addr 172.21.15.16:0/2301752163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda6d9d700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda81082b0 0x7efda819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda659c700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 0x7efda81a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 -- 172.21.15.16:0/2301752163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 msgr2=0x7efda81a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 0x7efda81a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 -- 172.21.15.16:0/2301752163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda81082b0 msgr2=0x7efda819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda81082b0 0x7efda819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda5d9b700 1 -- 172.21.15.16:0/2301752163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda819b030 con 0x7efda806b300 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.305+0000 7efda659c700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 0x7efda81a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:43.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.306+0000 7efda5d9b700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda81a0c50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c920 tx=0x7efd9000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.306+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd90007b00 con 0x7efda806b300 2024-04-25T01:26:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.306+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd90004ce0 con 0x7efda806b300 2024-04-25T01:26:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.306+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd90005550 con 0x7efda806b300 2024-04-25T01:26:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.306+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda819b320 con 0x7efda806b300 2024-04-25T01:26:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.306+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda81a75e0 con 0x7efda806b300 2024-04-25T01:26:43.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.307+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda8068f20 con 0x7efda806b300 2024-04-25T01:26:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.308+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efd90004750 con 0x7efda806b300 2024-04-25T01:26:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.309+0000 7efd9f7fe700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd940a2c20 0x7efd940a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.309+0000 7efda659c700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd940a2c20 0x7efd940a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.309+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7efd900cefb0 con 0x7efda806b300 2024-04-25T01:26:43.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.309+0000 7efda659c700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd940a2c20 0x7efd940a50e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd98005e20 tx=0x7efd98005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:43.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.313+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efd90014030 con 0x7efda806b300 2024-04-25T01:26:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.569+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7efda81a7870 con 0x7efda806b300 2024-04-25T01:26:43.549 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:43 smithi016 ceph-mon[29153]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:43.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.569+0000 7efd9f7fe700 1 -- 172.21.15.16:0/2301752163 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7efd9009be90 con 0x7efda806b300 2024-04-25T01:26:43.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:43.549 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd940a2c20 msgr2=0x7efd940a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd940a2c20 0x7efd940a50e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd98005e20 tx=0x7efd98005db0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 msgr2=0x7efda81a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda81a0c50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c920 tx=0x7efd9000cc30 comp rx=0 tx=0).stop 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 shutdown_connections 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7efd940a2c20 0x7efd940a50e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda81082b0 0x7efda819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806a960 0x7efda81a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 --2- 172.21.15.16:0/2301752163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda806b300 0x7efda81a0c50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:43.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 >> 172.21.15.16:0/2301752163 conn(0x7efda80780e0 msgr2=0x7efda80ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.572+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 shutdown_connections 2024-04-25T01:26:43.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:43.573+0000 7efdacce4700 1 -- 172.21.15.16:0/2301752163 wait complete. 2024-04-25T01:26:43.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:43 smithi038 ceph-mon[32841]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:43.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:43 smithi156 ceph-mon[32631]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:26:44.360 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:26:44.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:26:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.489+0000 7f44d845d640 1 -- 172.21.15.16:0/686761055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0101790 msgr2=0x7f44d0101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.489+0000 7f44d845d640 1 --2- 172.21.15.16:0/686761055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0101790 0x7f44d0101bb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009900 tx=0x7f44cc02a410 comp rx=0 tx=0).stop 2024-04-25T01:26:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- 172.21.15.16:0/686761055 shutdown_connections 2024-04-25T01:26:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 --2- 172.21.15.16:0/686761055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d0103340 0x7f44d0109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 --2- 172.21.15.16:0/686761055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 0x7f44d0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 --2- 172.21.15.16:0/686761055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0101790 0x7f44d0101bb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- 172.21.15.16:0/686761055 >> 172.21.15.16:0/686761055 conn(0x7f44d00fcd20 msgr2=0x7f44d00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- 172.21.15.16:0/686761055 shutdown_connections 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- 172.21.15.16:0/686761055 wait complete. 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 Processor -- start 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- start start 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d0101790 0x7f44d0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 0x7f44d0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 0x7f44d019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010d360 con 0x7f44d0103340 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010d1e0 con 0x7f44d0101790 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d845d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010d4e0 con 0x7f44d0102980 2024-04-25T01:26:44.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d69fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 0x7f44d019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d61f9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d0101790 0x7f44d0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d69fa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 0x7f44d019efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57872/0 (socket says 172.21.15.16:57872) 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.490+0000 7f44d69fa640 1 -- 172.21.15.16:0/3700254544 learned_addr learned my addr 172.21.15.16:0/3700254544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d59f8640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 0x7f44d0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d69fa640 1 -- 172.21.15.16:0/3700254544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 msgr2=0x7f44d0197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d69fa640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 0x7f44d0197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d69fa640 1 -- 172.21.15.16:0/3700254544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d0101790 msgr2=0x7f44d0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d69fa640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d0101790 0x7f44d0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d69fa640 1 -- 172.21.15.16:0/3700254544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d019f620 con 0x7f44d0103340 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d59f8640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 0x7f44d0197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d69fa640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 0x7f44d019efb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f44c800eb10 tx=0x7f44c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8004b20 con 0x7f44d0103340 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44c8004cc0 con 0x7f44d0103340 2024-04-25T01:26:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d019f910 con 0x7f44d0103340 2024-04-25T01:26:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8010690 con 0x7f44d0103340 2024-04-25T01:26:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.491+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d019fe90 con 0x7f44d0103340 2024-04-25T01:26:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.492+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d0108ab0 con 0x7f44d0103340 2024-04-25T01:26:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.493+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f44c8005000 con 0x7f44d0103340 2024-04-25T01:26:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.494+0000 7f44c77fe640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f44bc0a2b90 0x7f44bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.494+0000 7f44d61f9640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f44bc0a2b90 0x7f44bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.494+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f44c80cf870 con 0x7f44d0103340 2024-04-25T01:26:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.494+0000 7f44d61f9640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f44bc0a2b90 0x7f44bc0a5050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f44cc005ab0 tx=0x7f44cc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.498+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f44c8053740 con 0x7f44d0103340 2024-04-25T01:26:44.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:44 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2301752163' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:26:44.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.682+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44d0062340 con 0x7f44d0103340 2024-04-25T01:26:44.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.684+0000 7f44c77fe640 1 -- 172.21.15.16:0/3700254544 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7f44c809c7b0 con 0x7f44d0103340 2024-04-25T01:26:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:26:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f44bc0a2b90 msgr2=0x7f44bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f44bc0a2b90 0x7f44bc0a5050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f44cc005ab0 tx=0x7f44cc005a20 comp rx=0 tx=0).stop 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 msgr2=0x7f44d019efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 0x7f44d019efb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f44c800eb10 tx=0x7f44c800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 shutdown_connections 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f44bc0a2b90 0x7f44bc0a5050 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44d0103340 0x7f44d019efb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0102980 0x7f44d0197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 --2- 172.21.15.16:0/3700254544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44d0101790 0x7f44d0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 >> 172.21.15.16:0/3700254544 conn(0x7f44d00fcd20 msgr2=0x7f44d0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 shutdown_connections 2024-04-25T01:26:44.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.686+0000 7f44d845d640 1 -- 172.21.15.16:0/3700254544 wait complete. 2024-04-25T01:26:44.676 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:26:44.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:26:44.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:44 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2301752163' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:26:44.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:44 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2301752163' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 -- 172.21.15.16:0/227505310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0101310 msgr2=0x7fd7e0101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/227505310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0101310 0x7fd7e0101790 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009900 tx=0x7fd7d802a410 comp rx=0 tx=0).stop 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 -- 172.21.15.16:0/227505310 shutdown_connections 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/227505310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0101310 0x7fd7e0101790 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/227505310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7e0100970 0x7fd7e0100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/227505310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7e0100010 0x7fd7e0100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 -- 172.21.15.16:0/227505310 >> 172.21.15.16:0/227505310 conn(0x7fd7e00fbc30 msgr2=0x7fd7e00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 -- 172.21.15.16:0/227505310 shutdown_connections 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.805+0000 7fd7e7a46640 1 -- 172.21.15.16:0/227505310 wait complete. 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 Processor -- start 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 -- start start 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7e0100010 0x7fd7e0199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 0x7fd7e019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7e0101310 0x7fd7e01a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010f510 con 0x7fd7e0100970 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010f390 con 0x7fd7e0101310 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e7a46640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010f690 con 0x7fd7e0100010 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e4fe1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 0x7fd7e019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e4fe1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 0x7fd7e019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57884/0 (socket says 172.21.15.16:57884) 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e4fe1640 1 -- 172.21.15.16:0/2712664875 learned_addr learned my addr 172.21.15.16:0/2712664875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e57e2640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7e0100010 0x7fd7e0199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.806+0000 7fd7e5fe3640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7e0101310 0x7fd7e01a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e4fe1640 1 -- 172.21.15.16:0/2712664875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7e0100010 msgr2=0x7fd7e0199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e4fe1640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7e0100010 0x7fd7e0199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e4fe1640 1 -- 172.21.15.16:0/2712664875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7e0101310 msgr2=0x7fd7e01a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e4fe1640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7e0101310 0x7fd7e01a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e4fe1640 1 -- 172.21.15.16:0/2712664875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e01a18c0 con 0x7fd7e0100970 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e4fe1640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 0x7fd7e019a1c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000b4a0 tx=0x7fd7d000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0014070 con 0x7fd7e0100970 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d0010ac0 con 0x7fd7e0100970 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e01a1bb0 con 0x7fd7e0100970 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0017c30 con 0x7fd7e0100970 2024-04-25T01:26:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.807+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e01a2150 con 0x7fd7e0100970 2024-04-25T01:26:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.808+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e010ad60 con 0x7fd7e0100970 2024-04-25T01:26:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.809+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd7d0017070 con 0x7fd7e0100970 2024-04-25T01:26:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.810+0000 7fd7d67fc640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7cc0a2c10 0x7fd7cc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.810+0000 7fd7e57e2640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7cc0a2c10 0x7fd7cc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.810+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd7d00ce5e0 con 0x7fd7e0100970 2024-04-25T01:26:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.810+0000 7fd7e57e2640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7cc0a2c10 0x7fd7cc0a50d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd7e019ab50 tx=0x7fd7dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.815+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd7d009aed0 con 0x7fd7e0100970 2024-04-25T01:26:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.999+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd7e0062340 con 0x7fd7e0100970 2024-04-25T01:26:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:44.999+0000 7fd7d67fc640 1 -- 172.21.15.16:0/2712664875 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fd7d009f1d0 con 0x7fd7e0100970 2024-04-25T01:26:44.979 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:26:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7cc0a2c10 msgr2=0x7fd7cc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7cc0a2c10 0x7fd7cc0a50d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd7e019ab50 tx=0x7fd7dc009500 comp rx=0 tx=0).stop 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 msgr2=0x7fd7e019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 0x7fd7e019a1c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000b4a0 tx=0x7fd7d000b960 comp rx=0 tx=0).stop 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 shutdown_connections 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7cc0a2c10 0x7fd7cc0a50d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7e0101310 0x7fd7e01a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7e0100970 0x7fd7e019a1c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 --2- 172.21.15.16:0/2712664875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7e0100010 0x7fd7e0199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 >> 172.21.15.16:0/2712664875 conn(0x7fd7e00fbc30 msgr2=0x7fd7e00fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 shutdown_connections 2024-04-25T01:26:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.002+0000 7fd7e7a46640 1 -- 172.21.15.16:0/2712664875 wait complete. 2024-04-25T01:26:44.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 -- 172.21.15.16:0/666995499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 msgr2=0x7f44b0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/666995499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b0108ae0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009ca0 tx=0x7f44a802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 -- 172.21.15.16:0/666995499 shutdown_connections 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/666995499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b0108ae0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/666995499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 0x7f44b0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/666995499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 0x7f44b0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 -- 172.21.15.16:0/666995499 >> 172.21.15.16:0/666995499 conn(0x7f44b00fbc30 msgr2=0x7f44b00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 -- 172.21.15.16:0/666995499 shutdown_connections 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.118+0000 7f44b4e7b640 1 -- 172.21.15.16:0/666995499 wait complete. 2024-04-25T01:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 Processor -- start 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 -- start start 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 0x7f44b006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 0x7f44b019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b01a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44b010c1b0 con 0x7f44b0102250 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f44b010c030 con 0x7f44b01006a0 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44b4e7b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f44b010c330 con 0x7f44b0101890 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b01a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b01a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57900/0 (socket says 172.21.15.16:57900) 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44aed9d640 1 -- 172.21.15.16:0/3379443785 learned_addr learned my addr 172.21.15.16:0/3379443785 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44a5d9b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 0x7f44b019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.119+0000 7f44ae59c640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 0x7f44b006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44aed9d640 1 -- 172.21.15.16:0/3379443785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 msgr2=0x7f44b019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44aed9d640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 0x7f44b019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44aed9d640 1 -- 172.21.15.16:0/3379443785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 msgr2=0x7f44b006a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44aed9d640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 0x7f44b006a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44aed9d640 1 -- 172.21.15.16:0/3379443785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b01a17f0 con 0x7f44b0102250 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44a5d9b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 0x7f44b019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44ae59c640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 0x7f44b006a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44aed9d640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b01a1000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009dc0 tx=0x7f44a8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8032ec0 con 0x7f44b0102250 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a80044f0 con 0x7f44b0102250 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b01a1a20 con 0x7f44b0102250 2024-04-25T01:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8030d40 con 0x7f44b0102250 2024-04-25T01:26:45.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.120+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b01a1f70 con 0x7f44b0102250 2024-04-25T01:26:45.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.121+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b0107a30 con 0x7f44b0102250 2024-04-25T01:26:45.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.122+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f44a8004040 con 0x7f44b0102250 2024-04-25T01:26:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.123+0000 7f44a7fff640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f449c0a2d60 0x7f449c0a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.123+0000 7f44ae59c640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f449c0a2d60 0x7f449c0a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.123+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f44a80f0700 con 0x7f44b0102250 2024-04-25T01:26:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.124+0000 7f44ae59c640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f449c0a2d60 0x7f449c0a5220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f44a0005d00 tx=0x7f44a0005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.128+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f44a80bd640 con 0x7f44b0102250 2024-04-25T01:26:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.312+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44b002d350 con 0x7f44b0102250 2024-04-25T01:26:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.312+0000 7f44a7fff640 1 -- 172.21.15.16:0/3379443785 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f44a80bcf40 con 0x7f44b0102250 2024-04-25T01:26:45.292 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:26:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.314+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f449c0a2d60 msgr2=0x7f449c0a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.314+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f449c0a2d60 0x7f449c0a5220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f44a0005d00 tx=0x7f44a0005c90 comp rx=0 tx=0).stop 2024-04-25T01:26:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.314+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 msgr2=0x7f44b01a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.314+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b01a1000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009dc0 tx=0x7f44a8003ba0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 shutdown_connections 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f449c0a2d60 0x7f449c0a5220 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44b0102250 0x7f44b01a1000 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44b0101890 0x7f44b019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 --2- 172.21.15.16:0/3379443785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44b01006a0 0x7f44b006a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 >> 172.21.15.16:0/3379443785 conn(0x7f44b00fbc30 msgr2=0x7f44b0104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 shutdown_connections 2024-04-25T01:26:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.315+0000 7f44b4e7b640 1 -- 172.21.15.16:0/3379443785 wait complete. 2024-04-25T01:26:45.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:26:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 -- 172.21.15.16:0/3681639000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c103350 msgr2=0x7f678c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 --2- 172.21.15.16:0/3681639000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c103350 0x7f678c109be0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6784009ca0 tx=0x7f678402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 -- 172.21.15.16:0/3681639000 shutdown_connections 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 --2- 172.21.15.16:0/3681639000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c103350 0x7f678c109be0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 --2- 172.21.15.16:0/3681639000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c102990 0x7f678c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 --2- 172.21.15.16:0/3681639000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f678c1017a0 0x7f678c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 -- 172.21.15.16:0/3681639000 >> 172.21.15.16:0/3681639000 conn(0x7f678c0fcd10 msgr2=0x7f678c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 -- 172.21.15.16:0/3681639000 shutdown_connections 2024-04-25T01:26:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.437+0000 7f67908bc640 1 -- 172.21.15.16:0/3681639000 wait complete. 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 Processor -- start 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 -- start start 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c1017a0 0x7f678c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 0x7f678c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f678c103350 0x7f678c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f678c10d380 con 0x7f678c102990 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f678c10d200 con 0x7f678c1017a0 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f67908bc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f678c10d500 con 0x7f678c103350 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 0x7f678c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 0x7f678c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57928/0 (socket says 172.21.15.16:57928) 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 -- 172.21.15.16:0/1050423141 learned_addr learned my addr 172.21.15.16:0/1050423141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f678ad9d640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f678c103350 0x7f678c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 -- 172.21.15.16:0/1050423141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f678c103350 msgr2=0x7f678c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f678c103350 0x7f678c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f678a59c640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c1017a0 0x7f678c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 -- 172.21.15.16:0/1050423141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c1017a0 msgr2=0x7f678c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c1017a0 0x7f678c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f6783fff640 1 -- 172.21.15.16:0/1050423141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678c19f610 con 0x7f678c102990 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.438+0000 7f678a59c640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c1017a0 0x7f678c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.439+0000 7f6783fff640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 0x7f678c197f30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f677400b4a0 tx=0x7f677400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.439+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6774014070 con 0x7f678c102990 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.439+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6774010ac0 con 0x7f678c102990 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.439+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678c19f900 con 0x7f678c102990 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.439+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6774017c30 con 0x7f678c102990 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.439+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f678c19fe50 con 0x7f678c102990 2024-04-25T01:26:45.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.440+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f678c1089f0 con 0x7f678c102990 2024-04-25T01:26:45.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.441+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6774017070 con 0x7f678c102990 2024-04-25T01:26:45.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.442+0000 7f67837fe640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f67780a2b90 0x7f67780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.442+0000 7f678a59c640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f67780a2b90 0x7f67780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.442+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f67740ce7e0 con 0x7f678c102990 2024-04-25T01:26:45.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.442+0000 7f678a59c640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f67780a2b90 0x7f67780a5050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f678c1027f0 tx=0x7f677c005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:45.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.446+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f677409b720 con 0x7f678c102990 2024-04-25T01:26:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.630+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f678c062340 con 0x7f678c102990 2024-04-25T01:26:45.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:45 smithi016 ceph-mon[29153]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T01:26:45.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:45 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3700254544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:45.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:45 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2712664875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:26:45.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:45 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3379443785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:26:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.630+0000 7f67837fe640 1 -- 172.21.15.16:0/1050423141 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f677409b020 con 0x7f678c102990 2024-04-25T01:26:45.611 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f67780a2b90 msgr2=0x7f67780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f67780a2b90 0x7f67780a5050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f678c1027f0 tx=0x7f677c005db0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 msgr2=0x7f678c197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 0x7f678c197f30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f677400b4a0 tx=0x7f677400b960 comp rx=0 tx=0).stop 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 shutdown_connections 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f67780a2b90 0x7f67780a5050 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f678c103350 0x7f678c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c102990 0x7f678c197f30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 --2- 172.21.15.16:0/1050423141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f678c1017a0 0x7f678c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.632+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 >> 172.21.15.16:0/1050423141 conn(0x7f678c0fcd10 msgr2=0x7f678c105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:45.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.633+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 shutdown_connections 2024-04-25T01:26:45.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.633+0000 7f67908bc640 1 -- 172.21.15.16:0/1050423141 wait complete. 2024-04-25T01:26:45.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:26:45.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:45 smithi038 ceph-mon[32841]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T01:26:45.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:45 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3700254544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:45.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:45 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2712664875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:26:45.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:45 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3379443785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/767742295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 msgr2=0x7f0b78109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/767742295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b78109910 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009ca0 tx=0x7f0b7002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/767742295 shutdown_connections 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/767742295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b78109910 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/767742295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b78104bf0 0x7f0b78106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/767742295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b781022c0 0x7f0b781046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/767742295 >> 172.21.15.16:0/767742295 conn(0x7f0b780fbbf0 msgr2=0x7f0b780fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/767742295 shutdown_connections 2024-04-25T01:26:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.745+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/767742295 wait complete. 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 Processor -- start 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 -- start start 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b781022c0 0x7f0b781979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b78104bf0 0x7f0b78197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b7819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810d330 con 0x7f0b78107520 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810d1b0 con 0x7f0b781022c0 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ec6d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810d4b0 con 0x7f0b78104bf0 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b7819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b7819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57946/0 (socket says 172.21.15.16:57946) 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 -- 172.21.15.16:0/903283011 learned_addr learned my addr 172.21.15.16:0/903283011 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b6ffff640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b78104bf0 0x7f0b78197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ca09640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b781022c0 0x7f0b781979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 -- 172.21.15.16:0/903283011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b78104bf0 msgr2=0x7f0b78197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b78104bf0 0x7f0b78197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 -- 172.21.15.16:0/903283011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b781022c0 msgr2=0x7f0b781979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b781022c0 0x7f0b781979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7d20a640 1 -- 172.21.15.16:0/903283011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7819f600 con 0x7f0b78107520 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.746+0000 7f0b7ca09640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b781022c0 0x7f0b781979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.747+0000 7f0b7d20a640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b7819ef90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b70031040 tx=0x7f0b70003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.747+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70032ec0 con 0x7f0b78107520 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.747+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b700044f0 con 0x7f0b78107520 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.747+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7819f890 con 0x7f0b78107520 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.747+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70030cb0 con 0x7f0b78107520 2024-04-25T01:26:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.747+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7819fe30 con 0x7f0b78107520 2024-04-25T01:26:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.748+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b780fd6e0 con 0x7f0b78107520 2024-04-25T01:26:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.749+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0b70004040 con 0x7f0b78107520 2024-04-25T01:26:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.750+0000 7f0b6dffb640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0b600a2db0 0x7f0b600a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.750+0000 7f0b7ca09640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0b600a2db0 0x7f0b600a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.750+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0b700f06b0 con 0x7f0b78107520 2024-04-25T01:26:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.750+0000 7f0b7ca09640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0b600a2db0 0x7f0b600a5270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400baa0 tx=0x7f0b74005d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:45.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.754+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b700bd590 con 0x7f0b78107520 2024-04-25T01:26:45.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:45 smithi156 ceph-mon[32631]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T01:26:45.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:45 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3700254544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:45.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:45 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2712664875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:26:45.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:45 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3379443785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:26:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.978+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b78062340 con 0x7f0b78107520 2024-04-25T01:26:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.978+0000 7f0b6dffb640 1 -- 172.21.15.16:0/903283011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11866 (secure 0 0 0) 0x7f0b700bce90 con 0x7f0b78107520 2024-04-25T01:26:45.958 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:45.958 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:25:21.068325+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24476,24499,24524]},"id":1}]} 2024-04-25T01:26:45.958 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.980+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0b600a2db0 msgr2=0x7f0b600a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.980+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0b600a2db0 0x7f0b600a5270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400baa0 tx=0x7f0b74005d10 comp rx=0 tx=0).stop 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.980+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 msgr2=0x7f0b7819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.980+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b7819ef90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b70031040 tx=0x7f0b70003ba0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.980+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 shutdown_connections 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f0b600a2db0 0x7f0b600a5270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b78107520 0x7f0b7819ef90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b78104bf0 0x7f0b78197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 --2- 172.21.15.16:0/903283011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b781022c0 0x7f0b781979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 >> 172.21.15.16:0/903283011 conn(0x7f0b780fbbf0 msgr2=0x7f0b78102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 shutdown_connections 2024-04-25T01:26:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:45.981+0000 7f0b7ec6d640 1 -- 172.21.15.16:0/903283011 wait complete. 2024-04-25T01:26:45.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:26:46.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 -- 172.21.15.16:0/571845096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 msgr2=0x7fd250100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 --2- 172.21.15.16:0/571845096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250100aa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009900 tx=0x7fd24c02a410 comp rx=0 tx=0).stop 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 -- 172.21.15.16:0/571845096 shutdown_connections 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 --2- 172.21.15.16:0/571845096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd250102230 0x7fd250108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 --2- 172.21.15.16:0/571845096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250101870 0x7fd250101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 --2- 172.21.15.16:0/571845096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250100aa0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 -- 172.21.15.16:0/571845096 >> 172.21.15.16:0/571845096 conn(0x7fd2500fbbf0 msgr2=0x7fd2500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 -- 172.21.15.16:0/571845096 shutdown_connections 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.094+0000 7fd2580bc640 1 -- 172.21.15.16:0/571845096 wait complete. 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 Processor -- start 2024-04-25T01:26:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 -- start start 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd250101870 0x7fd250102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250102230 0x7fd250103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010c1b0 con 0x7fd250100680 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010c030 con 0x7fd250102230 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd2580bc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010c330 con 0x7fd250101870 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd255e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd255e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57970/0 (socket says 172.21.15.16:57970) 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd255e58640 1 -- 172.21.15.16:0/2691064138 learned_addr learned my addr 172.21.15.16:0/2691064138 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd256659640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250102230 0x7fd250103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.095+0000 7fd255657640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd250101870 0x7fd250102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd255e58640 1 -- 172.21.15.16:0/2691064138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd250101870 msgr2=0x7fd250102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd255e58640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd250101870 0x7fd250102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd255e58640 1 -- 172.21.15.16:0/2691064138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250102230 msgr2=0x7fd250103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd255e58640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250102230 0x7fd250103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd255e58640 1 -- 172.21.15.16:0/2691064138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd250103a20 con 0x7fd250100680 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd255e58640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250104610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0314e0 tx=0x7fd24c0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c03f030 con 0x7fd250100680 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2501a1a90 con 0x7fd250100680 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd24c003cb0 con 0x7fd250100680 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2501a2030 con 0x7fd250100680 2024-04-25T01:26:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.096+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c035070 con 0x7fd250100680 2024-04-25T01:26:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.097+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2501a2260 con 0x7fd250100680 2024-04-25T01:26:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.098+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd24c042030 con 0x7fd250100680 2024-04-25T01:26:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.099+0000 7fd246ffd640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd23c0a2b90 0x7fd23c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.099+0000 7fd255657640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd23c0a2b90 0x7fd23c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.099+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd24c0efb70 con 0x7fd250100680 2024-04-25T01:26:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.099+0000 7fd255657640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd23c0a2b90 0x7fd23c0a5050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd250104380 tx=0x7fd24000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.102+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd24c0bc400 con 0x7fd250100680 2024-04-25T01:26:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.286+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2501a2430 con 0x7fd250100680 2024-04-25T01:26:46.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.288+0000 7fd246ffd640 1 -- 172.21.15.16:0/2691064138 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7fd24c0c0700 con 0x7fd250100680 2024-04-25T01:26:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:46.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:26:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.289+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd23c0a2b90 msgr2=0x7fd23c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.289+0000 7fd2580bc640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd23c0a2b90 0x7fd23c0a5050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd250104380 tx=0x7fd24000b390 comp rx=0 tx=0).stop 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.289+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 msgr2=0x7fd250104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.289+0000 7fd2580bc640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250104610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0314e0 tx=0x7fd24c0042e0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 shutdown_connections 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd23c0a2b90 0x7fd23c0a5050 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250102230 0x7fd250103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd250101870 0x7fd250102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 --2- 172.21.15.16:0/2691064138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd250100680 0x7fd250104610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 >> 172.21.15.16:0/2691064138 conn(0x7fd2500fbbf0 msgr2=0x7fd2500689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 shutdown_connections 2024-04-25T01:26:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.290+0000 7fd2580bc640 1 -- 172.21.15.16:0/2691064138 wait complete. 2024-04-25T01:26:46.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:26:46.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.410+0000 7ff428ce1640 1 -- 172.21.15.16:0/1743086309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424101890 msgr2=0x7ff424101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.410+0000 7ff428ce1640 1 --2- 172.21.15.16:0/1743086309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424101890 0x7ff424101d10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009900 tx=0x7ff40c02a410 comp rx=0 tx=0).stop 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.410+0000 7ff428ce1640 1 -- 172.21.15.16:0/1743086309 shutdown_connections 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.410+0000 7ff428ce1640 1 --2- 172.21.15.16:0/1743086309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff424102250 0x7ff424108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.410+0000 7ff428ce1640 1 --2- 172.21.15.16:0/1743086309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424101890 0x7ff424101d10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.410+0000 7ff428ce1640 1 --2- 172.21.15.16:0/1743086309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4241006a0 0x7ff424100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- 172.21.15.16:0/1743086309 >> 172.21.15.16:0/1743086309 conn(0x7ff4240fbc30 msgr2=0x7ff4240fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- 172.21.15.16:0/1743086309 shutdown_connections 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- 172.21.15.16:0/1743086309 wait complete. 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 Processor -- start 2024-04-25T01:26:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- start start 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4241006a0 0x7ff424197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff424101890 0x7ff424198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 0x7ff42419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff42410c280 con 0x7ff424102250 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff42410c100 con 0x7ff4241006a0 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.411+0000 7ff428ce1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff42410c400 con 0x7ff424101890 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 0x7ff42419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 0x7ff42419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57980/0 (socket says 172.21.15.16:57980) 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 -- 172.21.15.16:0/3895616315 learned_addr learned my addr 172.21.15.16:0/3895616315 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff41bfff640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff424101890 0x7ff424198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff42259c640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4241006a0 0x7ff424197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 -- 172.21.15.16:0/3895616315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff424101890 msgr2=0x7ff424198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff424101890 0x7ff424198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 -- 172.21.15.16:0/3895616315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4241006a0 msgr2=0x7ff424197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4241006a0 0x7ff424197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 -- 172.21.15.16:0/3895616315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42419f6d0 con 0x7ff424102250 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff422d9d640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 0x7ff42419f060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00b970 tx=0x7ff41c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c019070 con 0x7ff424102250 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff41c014440 con 0x7ff424102250 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42419f9c0 con 0x7ff424102250 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0075d0 con 0x7ff424102250 2024-04-25T01:26:46.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.412+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42419ff10 con 0x7ff424102250 2024-04-25T01:26:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.413+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4241a0140 con 0x7ff424102250 2024-04-25T01:26:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.414+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff41c01c070 con 0x7ff424102250 2024-04-25T01:26:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.415+0000 7ff41b7fe640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4100a2b90 0x7ff4100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.415+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff41c0ce690 con 0x7ff424102250 2024-04-25T01:26:46.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.415+0000 7ff42259c640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4100a2b90 0x7ff4100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.416+0000 7ff42259c640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4100a2b90 0x7ff4100a5050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff4241016f0 tx=0x7ff41400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:46.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.420+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff41c09af20 con 0x7ff424102250 2024-04-25T01:26:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.604+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4241a03b0 con 0x7ff424102250 2024-04-25T01:26:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.605+0000 7ff41b7fe640 1 -- 172.21.15.16:0/3895616315 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7ff41c09f220 con 0x7ff424102250 2024-04-25T01:26:46.584 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:26:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4100a2b90 msgr2=0x7ff4100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4100a2b90 0x7ff4100a5050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff4241016f0 tx=0x7ff41400b390 comp rx=0 tx=0).stop 2024-04-25T01:26:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 msgr2=0x7ff42419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 0x7ff42419f060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00b970 tx=0x7ff41c00be30 comp rx=0 tx=0).stop 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 shutdown_connections 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4100a2b90 0x7ff4100a5050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff424102250 0x7ff42419f060 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff424101890 0x7ff424198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 --2- 172.21.15.16:0/3895616315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4241006a0 0x7ff424197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 >> 172.21.15.16:0/3895616315 conn(0x7ff4240fbc30 msgr2=0x7ff4240fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 shutdown_connections 2024-04-25T01:26:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.607+0000 7ff428ce1640 1 -- 172.21.15.16:0/3895616315 wait complete. 2024-04-25T01:26:46.596 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:46 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1050423141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:26:46.596 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:46 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/903283011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:46.596 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:46 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2691064138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:46.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:26:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 -- 172.21.15.16:0/2153977867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81022c0 msgr2=0x7f09a81046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 --2- 172.21.15.16:0/2153977867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81022c0 0x7f09a81046b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009900 tx=0x7f09a402a410 comp rx=0 tx=0).stop 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 -- 172.21.15.16:0/2153977867 shutdown_connections 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 --2- 172.21.15.16:0/2153977867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a8107520 0x7f09a8109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 --2- 172.21.15.16:0/2153977867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8104bf0 0x7f09a8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 --2- 172.21.15.16:0/2153977867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a81022c0 0x7f09a81046b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 -- 172.21.15.16:0/2153977867 >> 172.21.15.16:0/2153977867 conn(0x7f09a80fbc30 msgr2=0x7f09a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 -- 172.21.15.16:0/2153977867 shutdown_connections 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.723+0000 7f09af35a640 1 -- 172.21.15.16:0/2153977867 wait complete. 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 Processor -- start 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 -- start start 2024-04-25T01:26:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a81022c0 0x7f09a81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 0x7f09a8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8107520 0x7f09a819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a810d340 con 0x7f09a8104bf0 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09a810d1c0 con 0x7f09a8107520 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09af35a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09a810d4c0 con 0x7f09a81022c0 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 0x7f09a8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 0x7f09a8197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58004/0 (socket says 172.21.15.16:58004) 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 -- 172.21.15.16:0/4141829091 learned_addr learned my addr 172.21.15.16:0/4141829091 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ad0f6640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a81022c0 0x7f09a81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ad8f7640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8107520 0x7f09a819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 -- 172.21.15.16:0/4141829091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a81022c0 msgr2=0x7f09a81979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a81022c0 0x7f09a81979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 -- 172.21.15.16:0/4141829091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8107520 msgr2=0x7f09a819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8107520 0x7f09a819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ac8f5640 1 -- 172.21.15.16:0/4141829091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a819f610 con 0x7f09a8104bf0 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ad0f6640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a81022c0 0x7f09a81979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.724+0000 7f09ad8f7640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8107520 0x7f09a819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.725+0000 7f09ac8f5640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 0x7f09a8197f30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f099800c970 tx=0x7f099800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.725+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998014070 con 0x7f09a8104bf0 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.725+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0998004440 con 0x7f09a8104bf0 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.725+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a819f900 con 0x7f09a8104bf0 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.725+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998003e50 con 0x7f09a8104bf0 2024-04-25T01:26:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.725+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a819ff20 con 0x7f09a8104bf0 2024-04-25T01:26:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.726+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a80fd720 con 0x7f09a8104bf0 2024-04-25T01:26:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.727+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0998017070 con 0x7f09a8104bf0 2024-04-25T01:26:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.727+0000 7f099e7fc640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09940a2db0 0x7f09940a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.728+0000 7f09ad0f6640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09940a2db0 0x7f09940a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.728+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f09980cec70 con 0x7f09a8104bf0 2024-04-25T01:26:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.728+0000 7f09ad0f6640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09940a2db0 0x7f09940a5270 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09a40314e0 tx=0x7f09a4031d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:46.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.732+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f099801e030 con 0x7f09a8104bf0 2024-04-25T01:26:46.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:46 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1050423141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:26:46.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:46 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/903283011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:46.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:46 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2691064138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:46.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:46 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1050423141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:26:46.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:46 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/903283011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:46.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:46 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2691064138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.916+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f09a8062340 con 0x7f09a8104bf0 2024-04-25T01:26:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.917+0000 7f099e7fc640 1 -- 172.21.15.16:0/4141829091 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f099809bb50 con 0x7f09a8104bf0 2024-04-25T01:26:46.896 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:26:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09940a2db0 msgr2=0x7f09940a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09940a2db0 0x7f09940a5270 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09a40314e0 tx=0x7f09a4031d30 comp rx=0 tx=0).stop 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 msgr2=0x7f09a8197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 0x7f09a8197f30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f099800c970 tx=0x7f099800ce30 comp rx=0 tx=0).stop 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 shutdown_connections 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09940a2db0 0x7f09940a5270 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a8107520 0x7f09a819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a8104bf0 0x7f09a8197f30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 --2- 172.21.15.16:0/4141829091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a81022c0 0x7f09a81979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 >> 172.21.15.16:0/4141829091 conn(0x7f09a80fbc30 msgr2=0x7f09a8102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 shutdown_connections 2024-04-25T01:26:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:46.919+0000 7f09af35a640 1 -- 172.21.15.16:0/4141829091 wait complete. 2024-04-25T01:26:46.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:26:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 -- 172.21.15.16:0/3482258582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1000ffff0 msgr2=0x7fd100100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3482258582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1000ffff0 0x7fd100100410 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009900 tx=0x7fd0fc02a410 comp rx=0 tx=0).stop 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 -- 172.21.15.16:0/3482258582 shutdown_connections 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3482258582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1001012f0 0x7fd100101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3482258582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 0x7fd100100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3482258582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1000ffff0 0x7fd100100410 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 -- 172.21.15.16:0/3482258582 >> 172.21.15.16:0/3482258582 conn(0x7fd1000fbbf0 msgr2=0x7fd1000fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.034+0000 7fd107d7e640 1 -- 172.21.15.16:0/3482258582 shutdown_connections 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 -- 172.21.15.16:0/3482258582 wait complete. 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 Processor -- start 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 -- start start 2024-04-25T01:26:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1000ffff0 0x7fd100199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 0x7fd10019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 0x7fd1001a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010f490 con 0x7fd1001012f0 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010f310 con 0x7fd100100950 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd107d7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010f610 con 0x7fd1000ffff0 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd10631b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 0x7fd1001a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd10631b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 0x7fd1001a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58014/0 (socket says 172.21.15.16:58014) 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd10631b640 1 -- 172.21.15.16:0/3341245744 learned_addr learned my addr 172.21.15.16:0/3341245744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd105319640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 0x7fd10019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.035+0000 7fd105b1a640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1000ffff0 0x7fd100199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd10631b640 1 -- 172.21.15.16:0/3341245744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1000ffff0 msgr2=0x7fd100199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd10631b640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1000ffff0 0x7fd100199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd10631b640 1 -- 172.21.15.16:0/3341245744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 msgr2=0x7fd10019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd10631b640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 0x7fd10019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd10631b640 1 -- 172.21.15.16:0/3341245744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1001a1840 con 0x7fd1001012f0 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd105319640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 0x7fd10019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd10631b640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 0x7fd1001a11a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800eaa0 tx=0x7fd0f800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f800cdb0 con 0x7fd1001012f0 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1001a1b30 con 0x7fd1001012f0 2024-04-25T01:26:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0f8005020 con 0x7fd1001012f0 2024-04-25T01:26:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80116d0 con 0x7fd1001012f0 2024-04-25T01:26:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.036+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1001a20d0 con 0x7fd1001012f0 2024-04-25T01:26:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.037+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10010b070 con 0x7fd1001012f0 2024-04-25T01:26:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.038+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd0f80051d0 con 0x7fd1001012f0 2024-04-25T01:26:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.039+0000 7fd0f6ffd640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd0ec0a2b90 0x7fd0ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.039+0000 7fd105b1a640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd0ec0a2b90 0x7fd0ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.039+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd0f80cf660 con 0x7fd1001012f0 2024-04-25T01:26:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.039+0000 7fd105b1a640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd0ec0a2b90 0x7fd0ec0a5050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc000c00 tx=0x7fd0fc032ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.042+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd0f8016030 con 0x7fd1001012f0 2024-04-25T01:26:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.228+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd10002d170 con 0x7fd1001012f0 2024-04-25T01:26:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.228+0000 7fd0f6ffd640 1 -- 172.21.15.16:0/3341245744 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fd0f809c5a0 con 0x7fd1001012f0 2024-04-25T01:26:47.208 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd0ec0a2b90 msgr2=0x7fd0ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd0ec0a2b90 0x7fd0ec0a5050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc000c00 tx=0x7fd0fc032ba0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 msgr2=0x7fd1001a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 0x7fd1001a11a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800eaa0 tx=0x7fd0f800ef60 comp rx=0 tx=0).stop 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 shutdown_connections 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd0ec0a2b90 0x7fd0ec0a5050 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1001012f0 0x7fd1001a11a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100100950 0x7fd10019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 --2- 172.21.15.16:0/3341245744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1000ffff0 0x7fd100199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 >> 172.21.15.16:0/3341245744 conn(0x7fd1000fbbf0 msgr2=0x7fd1000fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.230+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 shutdown_connections 2024-04-25T01:26:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.231+0000 7fd107d7e640 1 -- 172.21.15.16:0/3341245744 wait complete. 2024-04-25T01:26:47.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:26:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.343+0000 7fb8f6134640 1 -- 172.21.15.16:0/1908208009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 msgr2=0x7fb8f0109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.343+0000 7fb8f6134640 1 --2- 172.21.15.16:0/1908208009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f0109be0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009ca0 tx=0x7fb8e802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- 172.21.15.16:0/1908208009 shutdown_connections 2024-04-25T01:26:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 --2- 172.21.15.16:0/1908208009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f0109be0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 --2- 172.21.15.16:0/1908208009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8f0102990 0x7fb8f0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 --2- 172.21.15.16:0/1908208009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 0x7fb8f0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- 172.21.15.16:0/1908208009 >> 172.21.15.16:0/1908208009 conn(0x7fb8f00fccf0 msgr2=0x7fb8f00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- 172.21.15.16:0/1908208009 shutdown_connections 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- 172.21.15.16:0/1908208009 wait complete. 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 Processor -- start 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- start start 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 0x7fb8f0197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8f0102990 0x7fb8f0197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010d2d0 con 0x7fb8f0103350 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010d150 con 0x7fb8f0102990 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.344+0000 7fb8f6134640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010d450 con 0x7fb8f01017a0 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58030/0 (socket says 172.21.15.16:58030) 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 -- 172.21.15.16:0/2552868221 learned_addr learned my addr 172.21.15.16:0/2552868221 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8ef7fe640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 0x7fb8f0197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8eeffd640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8f0102990 0x7fb8f0197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 -- 172.21.15.16:0/2552868221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 msgr2=0x7fb8f0197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 0x7fb8f0197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 -- 172.21.15.16:0/2552868221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8f0102990 msgr2=0x7fb8f0197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8f0102990 0x7fb8f0197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 -- 172.21.15.16:0/2552868221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f019f590 con 0x7fb8f0103350 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8ef7fe640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 0x7fb8f0197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8effff640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f019ef20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009dc0 tx=0x7fb8e8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8032ec0 con 0x7fb8f0103350 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e80044f0 con 0x7fb8f0103350 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f019f820 con 0x7fb8f0103350 2024-04-25T01:26:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8030d40 con 0x7fb8f0103350 2024-04-25T01:26:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.345+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8f019fd70 con 0x7fb8f0103350 2024-04-25T01:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.346+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8f0108b40 con 0x7fb8f0103350 2024-04-25T01:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.347+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb8e8004040 con 0x7fb8f0103350 2024-04-25T01:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.348+0000 7fb8ecff9640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb8d80a2d30 0x7fb8d80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.348+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb8e80f06a0 con 0x7fb8f0103350 2024-04-25T01:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.348+0000 7fb8ef7fe640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb8d80a2d30 0x7fb8d80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.349+0000 7fb8ef7fe640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb8d80a2d30 0x7fb8d80a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00058c0 tx=0x7fb8e0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.353+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb8e80bd580 con 0x7fb8f0103350 2024-04-25T01:26:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.576+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8f0062340 con 0x7fb8f0103350 2024-04-25T01:26:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.577+0000 7fb8ecff9640 1 -- 172.21.15.16:0/2552868221 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11866 (secure 0 0 0) 0x7fb8e80bce80 con 0x7fb8f0103350 2024-04-25T01:26:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:47.557 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:25:21.068325+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24476,24499,24524]},"id":1}]} 2024-04-25T01:26:47.557 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:26:47.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb8d80a2d30 msgr2=0x7fb8d80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb8d80a2d30 0x7fb8d80a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00058c0 tx=0x7fb8e0005850 comp rx=0 tx=0).stop 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 msgr2=0x7fb8f019ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f019ef20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009dc0 tx=0x7fb8e8003ba0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 shutdown_connections 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb8d80a2d30 0x7fb8d80a51f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0103350 0x7fb8f019ef20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8f0102990 0x7fb8f0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 --2- 172.21.15.16:0/2552868221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01017a0 0x7fb8f0197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 >> 172.21.15.16:0/2552868221 conn(0x7fb8f00fccf0 msgr2=0x7fb8f0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 shutdown_connections 2024-04-25T01:26:47.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.579+0000 7fb8f6134640 1 -- 172.21.15.16:0/2552868221 wait complete. 2024-04-25T01:26:47.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:47 smithi016 ceph-mon[29153]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T01:26:47.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:47 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3895616315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:47 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4141829091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:47 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3341245744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:26:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2265055106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c81017b0 msgr2=0x7f20c8101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2265055106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c81017b0 0x7f20c8101bd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009900 tx=0x7f20b802a410 comp rx=0 tx=0).stop 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2265055106 shutdown_connections 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2265055106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c8103360 0x7f20c8109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2265055106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81029a0 0x7f20c8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2265055106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c81017b0 0x7f20c8101bd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2265055106 >> 172.21.15.16:0/2265055106 conn(0x7f20c80fcd40 msgr2=0x7f20c80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2265055106 shutdown_connections 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2265055106 wait complete. 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 Processor -- start 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.693+0000 7f20cd6ee640 1 -- start start 2024-04-25T01:26:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81017b0 0x7f20c8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c81029a0 0x7f20c819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 0x7f20c81a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20c810d320 con 0x7f20c8103360 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f20c810d1a0 con 0x7f20c81029a0 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20c810d4a0 con 0x7f20c81017b0 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 0x7f20c81a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 0x7f20c81a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58054/0 (socket says 172.21.15.16:58054) 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 -- 172.21.15.16:0/2324169473 learned_addr learned my addr 172.21.15.16:0/2324169473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c67fc640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c81029a0 0x7f20c819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c6ffd640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81017b0 0x7f20c8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 -- 172.21.15.16:0/2324169473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81017b0 msgr2=0x7f20c8199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81017b0 0x7f20c8199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 -- 172.21.15.16:0/2324169473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c81029a0 msgr2=0x7f20c819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c81029a0 0x7f20c819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 -- 172.21.15.16:0/2324169473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c81a1890 con 0x7f20c8103360 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c6ffd640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81017b0 0x7f20c8199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20c77fe640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 0x7f20c81a11f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f20c000cad0 tx=0x7f20c000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c000ae20 con 0x7f20c8103360 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20c0005020 con 0x7f20c8103360 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c81a1b80 con 0x7f20c8103360 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c000f6b0 con 0x7f20c8103360 2024-04-25T01:26:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.694+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c81a2080 con 0x7f20c8103360 2024-04-25T01:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.696+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c81a2250 con 0x7f20c8103360 2024-04-25T01:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.697+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f20c00051d0 con 0x7f20c8103360 2024-04-25T01:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.697+0000 7f20affff640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f20b00a2b90 0x7f20b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.697+0000 7f20c6ffd640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f20b00a2b90 0x7f20b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.697+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f20c00cf190 con 0x7f20c8103360 2024-04-25T01:26:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.698+0000 7f20c6ffd640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f20b00a2b90 0x7f20b00a5050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20b8005ab0 tx=0x7f20b8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.702+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f20c0014030 con 0x7f20c8103360 2024-04-25T01:26:47.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:47 smithi038 ceph-mon[32841]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T01:26:47.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:47 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3895616315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:47 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/4141829091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:47 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3341245744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:47 smithi156 ceph-mon[32631]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T01:26:47.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:47 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3895616315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:47 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/4141829091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:47 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3341245744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:26:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.892+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20c81a2420 con 0x7f20c8103360 2024-04-25T01:26:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.893+0000 7f20affff640 1 -- 172.21.15.16:0/2324169473 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7f20c009c0d0 con 0x7f20c8103360 2024-04-25T01:26:47.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:47.873 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:26:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f20b00a2b90 msgr2=0x7f20b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f20b00a2b90 0x7f20b00a5050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20b8005ab0 tx=0x7f20b8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 msgr2=0x7f20c81a11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 0x7f20c81a11f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f20c000cad0 tx=0x7f20c000cf90 comp rx=0 tx=0).stop 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 shutdown_connections 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f20b00a2b90 0x7f20b00a5050 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20c8103360 0x7f20c81a11f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20c81029a0 0x7f20c819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 --2- 172.21.15.16:0/2324169473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20c81017b0 0x7f20c8199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 >> 172.21.15.16:0/2324169473 conn(0x7f20c80fcd40 msgr2=0x7f20c8073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 shutdown_connections 2024-04-25T01:26:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:47.896+0000 7f20cd6ee640 1 -- 172.21.15.16:0/2324169473 wait complete. 2024-04-25T01:26:47.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:26:47.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 -- 172.21.15.16:0/3416278185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 msgr2=0x7fe304108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 --2- 172.21.15.16:0/3416278185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe304108ae0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009ca0 tx=0x7fe2f402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 -- 172.21.15.16:0/3416278185 shutdown_connections 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 --2- 172.21.15.16:0/3416278185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe304108ae0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 --2- 172.21.15.16:0/3416278185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 0x7fe304101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 --2- 172.21.15.16:0/3416278185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3041006a0 0x7fe304100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 -- 172.21.15.16:0/3416278185 >> 172.21.15.16:0/3416278185 conn(0x7fe3040fbc30 msgr2=0x7fe3040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 -- 172.21.15.16:0/3416278185 shutdown_connections 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.009+0000 7fe308ef5640 1 -- 172.21.15.16:0/3416278185 wait complete. 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 Processor -- start 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 -- start start 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3041006a0 0x7fe30406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 0x7fe30419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe30419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe30410c1d0 con 0x7fe304102250 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe30410c050 con 0x7fe304101890 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe308ef5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe30410c350 con 0x7fe3041006a0 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe30419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe30419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58074/0 (socket says 172.21.15.16:58074) 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 -- 172.21.15.16:0/276456551 learned_addr learned my addr 172.21.15.16:0/276456551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe30259c640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3041006a0 0x7fe30406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe301d9b640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 0x7fe30419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 -- 172.21.15.16:0/276456551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3041006a0 msgr2=0x7fe30406a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3041006a0 0x7fe30406a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 -- 172.21.15.16:0/276456551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 msgr2=0x7fe30419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 0x7fe30419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:47.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe302d9d640 1 -- 172.21.15.16:0/276456551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30419f580 con 0x7fe304102250 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.010+0000 7fe301d9b640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 0x7fe30419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.011+0000 7fe302d9d640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe30419ecc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009dc0 tx=0x7fe2f4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.011+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4032ec0 con 0x7fe304102250 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.011+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2f40044f0 con 0x7fe304102250 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.011+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30419f810 con 0x7fe304102250 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.011+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4030d40 con 0x7fe304102250 2024-04-25T01:26:47.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.011+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30419fdb0 con 0x7fe304102250 2024-04-25T01:26:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.012+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe304107a60 con 0x7fe304102250 2024-04-25T01:26:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.013+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe2f4004040 con 0x7fe304102250 2024-04-25T01:26:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.014+0000 7fe2fb7fe640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe2f00a2db0 0x7fe2f00a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.014+0000 7fe30259c640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe2f00a2db0 0x7fe2f00a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.014+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe2f40f06f0 con 0x7fe304102250 2024-04-25T01:26:47.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.014+0000 7fe30259c640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe2f00a2db0 0x7fe2f00a5270 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc005fd0 tx=0x7fe2fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:47.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.018+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe2f40bd5d0 con 0x7fe304102250 2024-04-25T01:26:48.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.241+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe304062340 con 0x7fe304102250 2024-04-25T01:26:48.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.242+0000 7fe2fb7fe640 1 -- 172.21.15.16:0/276456551 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11866 (secure 0 0 0) 0x7fe2f40bced0 con 0x7fe304102250 2024-04-25T01:26:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:26:48.221 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:25:21.068325+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24476,24499,24524]},"id":1}]} 2024-04-25T01:26:48.222 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:26:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe2f00a2db0 msgr2=0x7fe2f00a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe2f00a2db0 0x7fe2f00a5270 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc005fd0 tx=0x7fe2fc005850 comp rx=0 tx=0).stop 2024-04-25T01:26:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 msgr2=0x7fe30419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe30419ecc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009dc0 tx=0x7fe2f4003ba0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 shutdown_connections 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe2f00a2db0 0x7fe2f00a5270 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe304102250 0x7fe30419ecc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe304101890 0x7fe30419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 --2- 172.21.15.16:0/276456551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3041006a0 0x7fe30406a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 >> 172.21.15.16:0/276456551 conn(0x7fe3040fbc30 msgr2=0x7fe304104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 shutdown_connections 2024-04-25T01:26:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.244+0000 7fe308ef5640 1 -- 172.21.15.16:0/276456551 wait complete. 2024-04-25T01:26:48.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:25:16.327985+0000', 'modified': '2024-04-25T01:25:21.068325+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24499, 'mds_2': 24476}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6835/2628472521', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6834', 'nonce': 2628472521}, {'type': 'v1', 'addr': '172.21.15.38:6835', 'nonce': 2628472521}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/2186765223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 2186765223}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 2186765223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/2504948433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 2504948433}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 2504948433}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'h', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.38:6837/401186797', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6836', 'nonce': 401186797}, {'type': 'v1', 'addr': '172.21.15.38:6837', 'nonce': 401186797}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.156:6839/1428073868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1428073868}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1428073868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/2790744598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 2790744598}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 2790744598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24476, 24499, 24524]} 2024-04-25T01:26:48.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T01:26:48.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:26:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 -- 172.21.15.16:0/4230089497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 msgr2=0x7fcf98101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 --2- 172.21.15.16:0/4230089497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf98101d10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009900 tx=0x7fcf8802a410 comp rx=0 tx=0).stop 2024-04-25T01:26:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 -- 172.21.15.16:0/4230089497 shutdown_connections 2024-04-25T01:26:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 --2- 172.21.15.16:0/4230089497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf98102250 0x7fcf98108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 --2- 172.21.15.16:0/4230089497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf98101d10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 --2- 172.21.15.16:0/4230089497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf981006a0 0x7fcf98100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 -- 172.21.15.16:0/4230089497 >> 172.21.15.16:0/4230089497 conn(0x7fcf980fbc30 msgr2=0x7fcf980fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 -- 172.21.15.16:0/4230089497 shutdown_connections 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 -- 172.21.15.16:0/4230089497 wait complete. 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 Processor -- start 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.359+0000 7fcfa0363640 1 -- start start 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf981006a0 0x7fcf9806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf9819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf98102250 0x7fcf9819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9810c240 con 0x7fcf98101890 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9810c0c0 con 0x7fcf98102250 2024-04-25T01:26:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9810c3c0 con 0x7fcf981006a0 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf9819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf9819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58092/0 (socket says 172.21.15.16:58092) 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/488009599 learned_addr learned my addr 172.21.15.16:0/488009599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf981006a0 0x7fcf9806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9e900640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf98102250 0x7fcf9819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/488009599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf981006a0 msgr2=0x7fcf9806a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf981006a0 0x7fcf9806a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/488009599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf98102250 msgr2=0x7fcf9819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf98102250 0x7fcf9819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/488009599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9819f6e0 con 0x7fcf98101890 2024-04-25T01:26:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf9819c7b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf880314e0 tx=0x7fcf88003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf8f7fe640 1 -- 172.21.15.16:0/488009599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8803f070 con 0x7fcf98101890 2024-04-25T01:26:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcf8f7fe640 1 -- 172.21.15.16:0/488009599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf88031e40 con 0x7fcf98101890 2024-04-25T01:26:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.360+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9819f910 con 0x7fcf98101890 2024-04-25T01:26:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.361+0000 7fcf8f7fe640 1 -- 172.21.15.16:0/488009599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf88035030 con 0x7fcf98101890 2024-04-25T01:26:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.361+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf9819fe00 con 0x7fcf98101890 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcf8f7fe640 1 -- 172.21.15.16:0/488009599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcf88042070 con 0x7fcf98101890 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 Processor -- start 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 -- start start 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e44930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e47430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559074da2f00 con 0x559074e3fe30 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559074d4b750 con 0x559074e44fa0 2024-04-25T01:26:48.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.363+0000 7fcfa306a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559074d9eec0 con 0x559074e3f440 2024-04-25T01:26:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e44930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e44930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58102/0 (socket says 172.21.15.16:58102) 2024-04-25T01:26:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/465732237 learned_addr learned my addr 172.21.15.16:0/465732237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf8f7fe640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcf840a2b90 0x7fcf840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e47430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e900640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcf840a2b90 0x7fcf840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e900640 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf8f7fe640 1 -- 172.21.15.16:0/488009599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcf880ef9b0 con 0x7fcf98101890 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/465732237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 msgr2=0x559074e3f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/465732237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 msgr2=0x559074e47430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e47430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/465732237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559074ab72d0 con 0x559074e3fe30 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e44930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400bcb0 tx=0x7fcf940107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad5e3ba555df93e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf9e900640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcf840a2b90 0x7fcf840a5050 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x559074e40690 tx=0x7fcf9000b3a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf8cbfe640 1 -- 172.21.15.16:0/465732237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf94005020 con 0x559074e3fe30 2024-04-25T01:26:48.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf8cbfe640 1 -- 172.21.15.16:0/465732237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf940047a0 con 0x559074e3fe30 2024-04-25T01:26:48.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.364+0000 7fcf8cbfe640 1 -- 172.21.15.16:0/465732237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf94011640 con 0x559074e3fe30 2024-04-25T01:26:48.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 -- 172.21.15.16:0/465732237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 msgr2=0x559074e44930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e44930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400bcb0 tx=0x7fcf940107a0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 -- 172.21.15.16:0/465732237 shutdown_connections 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e47430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e44930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 --2- 172.21.15.16:0/465732237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 -- 172.21.15.16:0/465732237 >> 172.21.15.16:0/465732237 conn(0x559074e3aab0 msgr2=0x559074e3aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 -- 172.21.15.16:0/465732237 shutdown_connections 2024-04-25T01:26:48.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 -- 172.21.15.16:0/465732237 wait complete. 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.365+0000 7fcfa306a740 1 -- start start 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 Processor -- start 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e498f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3fe30 0x559074e49e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e50e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559074e47f50 con 0x559074e3f440 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559074d5ecf0 con 0x559074e44fa0 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559074e480d0 con 0x559074e3fe30 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcfa306a740 10 client.0 _ll_register_callbacks cb 0x559074d5e6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58116/0 (socket says 172.21.15.16:58116) 2024-04-25T01:26:48.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/942627970 learned_addr learned my addr 172.21.15.16:0/942627970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3fe30 0x559074e49e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/942627970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3fe30 msgr2=0x559074e49e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3fe30 0x559074e49e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 msgr2=0x559074e50e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e50e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/942627970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559074b816b0 con 0x559074e3f440 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e498f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400bc80 tx=0x7fcf9400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4db60140997cf14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.366+0000 7fcf81ffb640 1 -- 172.21.15.16:0/942627970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9402b060 con 0x559074e3f440 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcf81ffb640 1 -- 172.21.15.16:0/942627970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcf94004f70 con 0x559074e3f440 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf940047a0 con 0x559074e3f440 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcf81ffb640 1 -- 172.21.15.16:0/942627970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcf94004c60 con 0x559074e3f440 2024-04-25T01:26:48.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 10 client.15048 fetch_fsmap learned FSMap version 21 2024-04-25T01:26:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559074e53de0 con 0x559074e3f440 2024-04-25T01:26:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcf81ffb640 1 -- 172.21.15.16:0/942627970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf940047a0 con 0x559074e3f440 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcf81ffb640 1 -- 172.21.15.16:0/942627970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcf94011cd0 con 0x559074e3f440 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 10 client.15048 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 10 client.15048 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 20 client.15048 populate_metadata read hostname 'smithi016' 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074e56130 0x559074e575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 4 client.15048 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.367+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 --> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559074db0440 con 0x559074e56130 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.368+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074e56130 0x559074e575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.368+0000 7fcf81ffb640 10 client.15048 ms_handle_connect on v2:172.21.15.38:6836/401186797 2024-04-25T01:26:48.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.368+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074e56130 0x559074e575a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1b705d906e87f35 server_cookie=444bee9e990f61f8 in_seq=0 out_seq=0 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcf81ffb640 1 -- 172.21.15.16:0/942627970 <== mds.2 v2:172.21.15.38:6836/401186797 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559074db0440 con 0x559074e56130 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcf81ffb640 10 client.15048 handle_command_reply: tid=0 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 client.15048 shutdown 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 msgr2=0x559074e498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e498f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400bc80 tx=0x7fcf9400bfd0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 shutdown_connections 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074e56130 0x559074e575a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e50e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3fe30 0x559074e49e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcf81ffb640 0 client.15048 ms_handle_reset on v2:172.21.15.38:6836/401186797 2024-04-25T01:26:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 --2- 172.21.15.16:0/942627970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e498f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 >> 172.21.15.16:0/942627970 conn(0x559074e3aab0 msgr2=0x559074e39ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 shutdown_connections 2024-04-25T01:26:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 1 -- 172.21.15.16:0/942627970 wait complete. 2024-04-25T01:26:48.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.369+0000 7fcfa306a740 20 client.15048 trim_cache size 0 max 16384 2024-04-25T01:26:48.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 Processor -- start 2024-04-25T01:26:48.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 -- start start 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e50d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e3d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559074e558e0 con 0x559074e3fe30 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559074e55fb0 con 0x559074e44fa0 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcfa306a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559074e56680 con 0x559074e3f440 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcf9d8fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e50d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcf9d8fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e50d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58128/0 (socket says 172.21.15.16:58128) 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/3158281600 learned_addr learned my addr 172.21.15.16:0/3158281600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.387+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/3158281600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 msgr2=0x559074e3ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/3158281600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 msgr2=0x559074e3d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e3d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9d8fe640 1 -- 172.21.15.16:0/3158281600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559074e3b780 con 0x559074e3fe30 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e50d20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcf88005c20 tx=0x7fcf8808e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28125444f66c5759 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf83fff640 1 -- 172.21.15.16:0/3158281600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf880efe60 con 0x559074e3fe30 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf83fff640 1 -- 172.21.15.16:0/3158281600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf880bc1f0 con 0x559074e3fe30 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcf83fff640 1 -- 172.21.15.16:0/3158281600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf88049040 con 0x559074e3fe30 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 -- 172.21.15.16:0/3158281600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 msgr2=0x559074e50d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e50d20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcf88005c20 tx=0x7fcf8808e200 comp rx=0 tx=0).stop 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 -- 172.21.15.16:0/3158281600 shutdown_connections 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e44fa0 0x559074e3d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3fe30 0x559074e50d20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 --2- 172.21.15.16:0/3158281600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e3f440 0x559074e3ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 -- 172.21.15.16:0/3158281600 >> 172.21.15.16:0/3158281600 conn(0x559074e3aab0 msgr2=0x559074e35690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 -- 172.21.15.16:0/3158281600 shutdown_connections 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.388+0000 7fcfa306a740 1 -- 172.21.15.16:0/3158281600 wait complete. 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 -- start start 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 Processor -- start 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e4ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e3fe30 0x559074e4f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e44fa0 0x559074e4f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559074e57420 con 0x559074e3f440 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559074e56d50 con 0x559074e3fe30 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559074e57af0 con 0x559074e44fa0 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcfa306a740 10 client.0 _ll_register_callbacks cb 0x7fcf9400b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e4ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e4ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58140/0 (socket says 172.21.15.16:58140) 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/1897046667 learned_addr learned my addr 172.21.15.16:0/1897046667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e900640 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e44fa0 0x559074e4f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/1897046667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e44fa0 msgr2=0x559074e4f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e44fa0 0x559074e4f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e3fe30 msgr2=0x559074e4f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e3fe30 0x559074e4f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.389+0000 7fcf9e0ff640 1 -- 172.21.15.16:0/1897046667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559074e50150 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf9e0ff640 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e4ef60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcf94011010 tx=0x7fcf94005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6ee4bbad1bb6bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf81ffb640 1 -- 172.21.15.16:0/1897046667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9402b070 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf81ffb640 1 -- 172.21.15.16:0/1897046667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcf9401bd30 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf880bc1f0 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf81ffb640 1 -- 172.21.15.16:0/1897046667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcf94004f20 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 10 client.15054 fetch_fsmap learned FSMap version 21 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559074ebc360 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf81ffb640 1 -- 172.21.15.16:0/1897046667 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf880bc1f0 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcf81ffb640 1 -- 172.21.15.16:0/1897046667 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcf94026890 con 0x559074e3f440 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 10 client.15054 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:26:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 10 client.15054 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 20 client.15054 populate_metadata read hostname 'smithi016' 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074ec0fc0 0x559074ebfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 4 client.15054 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.390+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 --> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] -- command(tid 0: {"prefix": "status"}) -- 0x559074daa790 con 0x559074ec0fc0 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074ec0fc0 0x559074ebfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcf9d8fe640 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074ec0fc0 0x559074ebfcd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c24c22c21fcea9de server_cookie=21078fcd6ac61d6f in_seq=0 out_seq=0 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcf81ffb640 10 client.15054 ms_handle_connect on v2:172.21.15.38:6836/401186797 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcf81ffb640 1 -- 172.21.15.16:0/1897046667 <== mds.2 v2:172.21.15.38:6836/401186797 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559074daa790 con 0x559074ec0fc0 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcf81ffb640 10 client.15054 handle_command_reply: tid=0 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 client.15054 shutdown 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 msgr2=0x559074e4ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e4ef60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcf94011010 tx=0x7fcf94005cc0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 shutdown_connections 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x559074ec0fc0 0x559074ebfcd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559074e44fa0 0x559074e4f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcf81ffb640 0 client.15054 ms_handle_reset on v2:172.21.15.38:6836/401186797 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559074e3fe30 0x559074e4f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 --2- 172.21.15.16:0/1897046667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559074e3f440 0x559074e4ef60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 >> 172.21.15.16:0/1897046667 conn(0x559074e3aab0 msgr2=0x559074e3ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 shutdown_connections 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 1 -- 172.21.15.16:0/1897046667 wait complete. 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.391+0000 7fcfa306a740 20 client.15054 trim_cache size 0 max 16384 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24476, 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-25T01:26:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-25T01:26:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 88.328624145999996, 2024-04-25T01:26:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 21, 2024-04-25T01:26:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 87, 2024-04-25T01:26:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-04-25T01:26:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 109.353082122 2024-04-25T01:26:48.381 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:26:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcf840a2b90 msgr2=0x7fcf840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcf840a2b90 0x7fcf840a5050 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x559074e40690 tx=0x7fcf9000b3a0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 msgr2=0x7fcf9819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf9819c7b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf880314e0 tx=0x7fcf88003c80 comp rx=0 tx=0).stop 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 shutdown_connections 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcf840a2b90 0x7fcf840a5050 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf98102250 0x7fcf9819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf98101890 0x7fcf9819c7b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 --2- 172.21.15.16:0/488009599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf981006a0 0x7fcf9806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 >> 172.21.15.16:0/488009599 conn(0x7fcf980fbc30 msgr2=0x7fcf98104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 shutdown_connections 2024-04-25T01:26:48.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.392+0000 7fcfa0363640 1 -- 172.21.15.16:0/488009599 wait complete. 2024-04-25T01:26:48.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:26:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 -- 172.21.15.16:0/764102778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 msgr2=0x7faa3c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 --2- 172.21.15.16:0/764102778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c101bc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa34009900 tx=0x7faa3402a410 comp rx=0 tx=0).stop 2024-04-25T01:26:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 -- 172.21.15.16:0/764102778 shutdown_connections 2024-04-25T01:26:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 --2- 172.21.15.16:0/764102778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa3c103350 0x7faa3c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 --2- 172.21.15.16:0/764102778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c102990 0x7faa3c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 --2- 172.21.15.16:0/764102778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c101bc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 -- 172.21.15.16:0/764102778 >> 172.21.15.16:0/764102778 conn(0x7faa3c0fcd10 msgr2=0x7faa3c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 -- 172.21.15.16:0/764102778 shutdown_connections 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.508+0000 7faa41314640 1 -- 172.21.15.16:0/764102778 wait complete. 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 Processor -- start 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 -- start start 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c102990 0x7faa3c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa3c103350 0x7faa3c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10d2c0 con 0x7faa3c1017a0 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10d140 con 0x7faa3c103350 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa41314640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10d440 con 0x7faa3c102990 2024-04-25T01:26:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58158/0 (socket says 172.21.15.16:58158) 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa3affd640 1 -- 172.21.15.16:0/1012694653 learned_addr learned my addr 172.21.15.16:0/1012694653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.509+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa3c103350 0x7faa3c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c102990 0x7faa3c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3affd640 1 -- 172.21.15.16:0/1012694653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c102990 msgr2=0x7faa3c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3affd640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c102990 0x7faa3c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3affd640 1 -- 172.21.15.16:0/1012694653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa3c103350 msgr2=0x7faa3c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3affd640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa3c103350 0x7faa3c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3affd640 1 -- 172.21.15.16:0/1012694653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c19f580 con 0x7faa3c1017a0 2024-04-25T01:26:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa3affd640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c1979b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faa340314e0 tx=0x7faa340042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa33fff640 1 -- 172.21.15.16:0/1012694653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3403f030 con 0x7faa3c1017a0 2024-04-25T01:26:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa33fff640 1 -- 172.21.15.16:0/1012694653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa34003cb0 con 0x7faa3c1017a0 2024-04-25T01:26:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3c19f810 con 0x7faa3c1017a0 2024-04-25T01:26:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa33fff640 1 -- 172.21.15.16:0/1012694653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34035070 con 0x7faa3c1017a0 2024-04-25T01:26:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.510+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3c19fd00 con 0x7faa3c1017a0 2024-04-25T01:26:48.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa33fff640 1 -- 172.21.15.16:0/1012694653 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faa34042030 con 0x7faa3c1017a0 2024-04-25T01:26:48.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 Processor -- start 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 -- start start 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a29850 0x560ad9a29c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 0x560ad9a2ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a31840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ad9988f00 con 0x560ad9a2a240 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ad9931750 con 0x560ad9a2f3b0 2024-04-25T01:26:48.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.513+0000 7faa44029740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ad9984ec0 con 0x560ad9a29850 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 0x560ad9a2ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 0x560ad9a2ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58166/0 (socket says 172.21.15.16:58166) 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 -- 172.21.15.16:0/1979706281 learned_addr learned my addr 172.21.15.16:0/1979706281 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a31840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a29850 0x560ad9a29c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 -- 172.21.15.16:0/1979706281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a29850 msgr2=0x560ad9a29c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a29850 0x560ad9a29c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 -- 172.21.15.16:0/1979706281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 msgr2=0x560ad9a31840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a31840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 -- 172.21.15.16:0/1979706281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ad969d2d0 con 0x560ad9a2a240 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa33fff640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa280a2b90 0x7faa280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3affd640 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 0x560ad9a2ed40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faa340f5c10 tx=0x7faa340f5d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b373e7822f11c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa280a2b90 0x7faa280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa33fff640 1 -- 172.21.15.16:0/1012694653 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa340ef680 con 0x7faa3c1017a0 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa317fa640 1 -- 172.21.15.16:0/1979706281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340b8160 con 0x560ad9a2a240 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa317fa640 1 -- 172.21.15.16:0/1979706281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa340bc5c0 con 0x560ad9a2a240 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.514+0000 7faa317fa640 1 -- 172.21.15.16:0/1979706281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340bb820 con 0x560ad9a2a240 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 -- 172.21.15.16:0/1979706281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 msgr2=0x560ad9a2ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa280a2b90 0x7faa280a5050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x560ad9a2aaa0 tx=0x7faa2c007520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 0x560ad9a2ed40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faa340f5c10 tx=0x7faa340f5d00 comp rx=0 tx=0).stop 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 -- 172.21.15.16:0/1979706281 shutdown_connections 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a31840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2a240 0x560ad9a2ed40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 --2- 172.21.15.16:0/1979706281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a29850 0x560ad9a29c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 -- 172.21.15.16:0/1979706281 >> 172.21.15.16:0/1979706281 conn(0x560ad9a20ab0 msgr2=0x560ad9a20eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 -- 172.21.15.16:0/1979706281 shutdown_connections 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 -- 172.21.15.16:0/1979706281 wait complete. 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.515+0000 7faa44029740 1 -- start start 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 Processor -- start 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a33a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a33f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a3afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a32120 con 0x560ad9a29850 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ad9944cf0 con 0x560ad9a2f3b0 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a322a0 con 0x560ad9a2a240 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 10 client.0 _ll_register_callbacks cb 0x560ad99446e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a33a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a33a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58170/0 (socket says 172.21.15.16:58170) 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 -- 172.21.15.16:0/1611649479 learned_addr learned my addr 172.21.15.16:0/1611649479 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a3afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a33f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 -- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 msgr2=0x560ad9a33f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a33f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 -- 172.21.15.16:0/1611649479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 msgr2=0x560ad9a3afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a3afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 -- 172.21.15.16:0/1611649479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ad97676b0 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a33f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa3affd640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a33a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faa340bbbc0 tx=0x7faa34004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b37960e900d09f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa227fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa227fc640 1 -- 172.21.15.16:0/1611649479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34049040 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa227fc640 1 -- 172.21.15.16:0/1611649479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa3403bc90 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.516+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa340bc5c0 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa227fc640 1 -- 172.21.15.16:0/1611649479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa340bba00 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 10 client.15066 fetch_fsmap learned FSMap version 21 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560ad9a32120 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa227fc640 1 -- 172.21.15.16:0/1611649479 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa340bc5c0 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa227fc640 1 -- 172.21.15.16:0/1611649479 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7faa3403b460 con 0x560ad9a29850 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 10 client.15066 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:26:48.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 10 client.15066 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 20 client.15066 populate_metadata read hostname 'smithi016' 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9a40380 0x560ad9a417f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 4 client.15066 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 --> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ad9996440 con 0x560ad9a40380 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.517+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9a40380 0x560ad9a417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.518+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9a40380 0x560ad9a417f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7120ff6f16415f5f server_cookie=a7eacf2fb1e7eba9 in_seq=0 out_seq=0 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.518+0000 7faa227fc640 10 client.15066 ms_handle_connect on v2:172.21.15.156:6838/1428073868 2024-04-25T01:26:48.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa227fc640 1 -- 172.21.15.16:0/1611649479 <== mds.1 v2:172.21.15.156:6838/1428073868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ad9996440 con 0x560ad9a40380 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa227fc640 10 client.15066 handle_command_reply: tid=0 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 client.15066 shutdown 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 msgr2=0x560ad9a33a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a33a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faa340bbbc0 tx=0x7faa34004590 comp rx=0 tx=0).stop 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 shutdown_connections 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9a40380 0x560ad9a417f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2f3b0 0x560ad9a3afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a33f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa227fc640 0 client.15066 ms_handle_reset on v2:172.21.15.156:6838/1428073868 2024-04-25T01:26:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 --2- 172.21.15.16:0/1611649479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a33a50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 >> 172.21.15.16:0/1611649479 conn(0x560ad9a20ab0 msgr2=0x560ad9a1fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 shutdown_connections 2024-04-25T01:26:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 1 -- 172.21.15.16:0/1611649479 wait complete. 2024-04-25T01:26:48.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.519+0000 7faa44029740 20 client.15066 trim_cache size 0 max 16384 2024-04-25T01:26:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 Processor -- start 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 -- start start 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a29850 0x560ad9a20260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a3af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 0x560ad9a32d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a3fb30 con 0x560ad9a2f3b0 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a40200 con 0x560ad9a29850 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa44029740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a408d0 con 0x560ad9a2a240 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.537+0000 7faa3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 0x560ad9a32d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 0x560ad9a32d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58186/0 (socket says 172.21.15.16:58186) 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 -- 172.21.15.16:0/572901235 learned_addr learned my addr 172.21.15.16:0/572901235 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a3af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3affd640 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a29850 0x560ad9a20260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 -- 172.21.15.16:0/572901235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 msgr2=0x560ad9a3af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a3af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 -- 172.21.15.16:0/572901235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a29850 msgr2=0x560ad9a20260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a29850 0x560ad9a20260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 -- 172.21.15.16:0/572901235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ad9a2a620 con 0x560ad9a2f3b0 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 0x560ad9a32d20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7faa2c01d880 tx=0x7faa2c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b082e5ee01a5e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa30ff9640 1 -- 172.21.15.16:0/572901235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c003970 con 0x560ad9a2f3b0 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa30ff9640 1 -- 172.21.15.16:0/572901235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2c00f9a0 con 0x560ad9a2f3b0 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.538+0000 7faa30ff9640 1 -- 172.21.15.16:0/572901235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c004500 con 0x560ad9a2f3b0 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- 172.21.15.16:0/572901235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 msgr2=0x560ad9a32d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 0x560ad9a32d20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7faa2c01d880 tx=0x7faa2c018000 comp rx=0 tx=0).stop 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- 172.21.15.16:0/572901235 shutdown_connections 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a2f3b0 0x560ad9a32d20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2a240 0x560ad9a3af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- 172.21.15.16:0/572901235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a29850 0x560ad9a20260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- 172.21.15.16:0/572901235 >> 172.21.15.16:0/572901235 conn(0x560ad9a20ab0 msgr2=0x560ad9a24250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- 172.21.15.16:0/572901235 shutdown_connections 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- 172.21.15.16:0/572901235 wait complete. 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- start start 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 Processor -- start 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a3a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2a240 0x560ad9a3ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2f3b0 0x560ad9a372c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a41670 con 0x560ad9a29850 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a40fa0 con 0x560ad9a2a240 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.539+0000 7faa44029740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ad9a41d40 con 0x560ad9a2f3b0 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa44029740 10 client.0 _ll_register_callbacks cb 0x7faa340f5a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a3a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a3a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58192/0 (socket says 172.21.15.16:58192) 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 -- 172.21.15.16:0/1606753096 learned_addr learned my addr 172.21.15.16:0/1606753096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2a240 0x560ad9a3ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 -- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2f3b0 msgr2=0x560ad9a372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2f3b0 0x560ad9a372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 -- 172.21.15.16:0/1606753096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2a240 msgr2=0x560ad9a3ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2a240 0x560ad9a3ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 -- 172.21.15.16:0/1606753096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ad9a37960 con 0x560ad9a29850 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2a240 0x560ad9a3ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa3affd640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a3a7d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faa340bc1b0 tx=0x7faa3402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d203c310a2386d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa227fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa227fc640 1 -- 172.21.15.16:0/1606753096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34049040 con 0x560ad9a29850 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa227fc640 1 -- 172.21.15.16:0/1606753096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa3403bc90 con 0x560ad9a29850 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa2c00f9a0 con 0x560ad9a29850 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa227fc640 1 -- 172.21.15.16:0/1606753096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa340bbc90 con 0x560ad9a29850 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa44029740 10 client.15072 fetch_fsmap learned FSMap version 21 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560ad9a37960 con 0x560ad9a29850 2024-04-25T01:26:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa227fc640 1 -- 172.21.15.16:0/1606753096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2c00f9a0 con 0x560ad9a29850 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.540+0000 7faa227fc640 1 -- 172.21.15.16:0/1606753096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7faa340c46a0 con 0x560ad9a29850 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa44029740 10 client.15072 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa44029740 10 client.15072 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa44029740 20 client.15072 populate_metadata read hostname 'smithi016' 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa44029740 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9aab7f0 0x560ad9aaa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa44029740 4 client.15072 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 --> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] -- command(tid 0: {"prefix": "status"}) -- 0x560ad9990790 con 0x560ad9aab7f0 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9aab7f0 0x560ad9aaa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa3a7fc640 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9aab7f0 0x560ad9aaa500 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0365eb60231e8ac server_cookie=5f1cc82f38c6e9e in_seq=0 out_seq=0 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.541+0000 7faa227fc640 10 client.15072 ms_handle_connect on v2:172.21.15.156:6838/1428073868 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa227fc640 1 -- 172.21.15.16:0/1606753096 <== mds.1 v2:172.21.15.156:6838/1428073868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560ad9990790 con 0x560ad9aab7f0 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa227fc640 10 client.15072 handle_command_reply: tid=0 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 client.15072 shutdown 2024-04-25T01:26:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 msgr2=0x560ad9a3a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a3a7d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faa340bc1b0 tx=0x7faa3402a410 comp rx=0 tx=0).stop 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 shutdown_connections 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x560ad9aab7f0 0x560ad9aaa500 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ad9a2f3b0 0x560ad9a372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa227fc640 0 client.15072 ms_handle_reset on v2:172.21.15.156:6838/1428073868 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ad9a2a240 0x560ad9a3ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 --2- 172.21.15.16:0/1606753096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560ad9a29850 0x560ad9a3a7d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 >> 172.21.15.16:0/1606753096 conn(0x560ad9a20ab0 msgr2=0x560ad9a1f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 shutdown_connections 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 1 -- 172.21.15.16:0/1606753096 wait complete. 2024-04-25T01:26:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.542+0000 7faa44029740 20 client.15072 trim_cache size 0 max 16384 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24499, 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 89.485150560999998, 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 21, 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 87, 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 106.615458299 2024-04-25T01:26:48.530 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:26:48.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa280a2b90 msgr2=0x7faa280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa280a2b90 0x7faa280a5050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x560ad9a2aaa0 tx=0x7faa2c007520 comp rx=0 tx=0).stop 2024-04-25T01:26:48.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 msgr2=0x7faa3c1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c1979b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faa340314e0 tx=0x7faa340042e0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 shutdown_connections 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa280a2b90 0x7faa280a5050 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa3c103350 0x7faa3c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c102990 0x7faa3c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 --2- 172.21.15.16:0/1012694653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c1017a0 0x7faa3c1979b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 >> 172.21.15.16:0/1012694653 conn(0x7faa3c0fcd10 msgr2=0x7faa3c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 shutdown_connections 2024-04-25T01:26:48.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.543+0000 7faa41314640 1 -- 172.21.15.16:0/1012694653 wait complete. 2024-04-25T01:26:48.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:48 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2552868221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:48 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2324169473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:48 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/276456551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:26:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 -- 172.21.15.16:0/1221224031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68104b50 msgr2=0x7f7b68106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/1221224031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68104b50 0x7f7b68106f40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009900 tx=0x7f7b5c02a410 comp rx=0 tx=0).stop 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 -- 172.21.15.16:0/1221224031 shutdown_connections 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/1221224031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68107480 0x7f7b681098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/1221224031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68104b50 0x7f7b68106f40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/1221224031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b68102220 0x7f7b68104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 -- 172.21.15.16:0/1221224031 >> 172.21.15.16:0/1221224031 conn(0x7f7b680fbc10 msgr2=0x7f7b680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 -- 172.21.15.16:0/1221224031 shutdown_connections 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.656+0000 7f7b6de6b640 1 -- 172.21.15.16:0/1221224031 wait complete. 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 Processor -- start 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 -- start start 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 0x7f7b68101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b68104b50 0x7f7b681001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68107480 0x7f7b68100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6810d320 con 0x7f7b68102220 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6810d1a0 con 0x7f7b68104b50 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b6de6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6810d4a0 con 0x7f7b68107480 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 0x7f7b68101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 0x7f7b68101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58210/0 (socket says 172.21.15.16:58210) 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b677fe640 1 -- 172.21.15.16:0/2419859729 learned_addr learned my addr 172.21.15.16:0/2419859729 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.657+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b68104b50 0x7f7b681001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b67fff640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68107480 0x7f7b68100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b677fe640 1 -- 172.21.15.16:0/2419859729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68107480 msgr2=0x7f7b68100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b677fe640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68107480 0x7f7b68100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b677fe640 1 -- 172.21.15.16:0/2419859729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b68104b50 msgr2=0x7f7b681001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b677fe640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b68104b50 0x7f7b681001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b677fe640 1 -- 172.21.15.16:0/2419859729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b68100fa0 con 0x7f7b68102220 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b677fe640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 0x7f7b68101b90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800d950 tx=0x7f7b5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b64ff9640 1 -- 172.21.15.16:0/2419859729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58017030 con 0x7f7b68102220 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b64ff9640 1 -- 172.21.15.16:0/2419859729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b58004020 con 0x7f7b68102220 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b681a1b40 con 0x7f7b68102220 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b64ff9640 1 -- 172.21.15.16:0/2419859729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580047a0 con 0x7f7b68102220 2024-04-25T01:26:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.658+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b681a20b0 con 0x7f7b68102220 2024-04-25T01:26:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.660+0000 7f7b64ff9640 1 -- 172.21.15.16:0/2419859729 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7b580041d0 con 0x7f7b68102220 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 Processor -- start 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 -- start start 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947440 0x55b1cf947860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf94c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf94f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf8aaf00 con 0x55b1cf947e30 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf853750 con 0x55b1cf947440 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b70b79740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf8a6ec0 con 0x55b1cf94cfa0 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf94c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf94c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58216/0 (socket says 172.21.15.16:58216) 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 -- 172.21.15.16:0/3877582954 learned_addr learned my addr 172.21.15.16:0/3877582954 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b64ff9640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7b500a2c10 0x7f7b500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf94f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b67fff640 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947440 0x55b1cf947860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b67fff640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7b500a2c10 0x7f7b500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b64ff9640 1 -- 172.21.15.16:0/2419859729 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7b580ceb40 con 0x7f7b68102220 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 -- 172.21.15.16:0/3877582954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 msgr2=0x55b1cf94f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf94f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 -- 172.21.15.16:0/3877582954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947440 msgr2=0x55b1cf947860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947440 0x55b1cf947860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b677fe640 1 -- 172.21.15.16:0/3877582954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1cf5bf2d0 con 0x55b1cf947e30 2024-04-25T01:26:48.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.661+0000 7f7b67fff640 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947440 0x55b1cf947860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b677fe640 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf94c930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7b5809b080 tx=0x7f7b5809aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e561da1ee0ea65bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b67fff640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7b500a2c10 0x7f7b500a50d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55b1cf948690 tx=0x7f7b60009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b4dffb640 1 -- 172.21.15.16:0/3877582954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580d2190 con 0x55b1cf947e30 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b4dffb640 1 -- 172.21.15.16:0/3877582954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b5809ba80 con 0x55b1cf947e30 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b4dffb640 1 -- 172.21.15.16:0/3877582954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58009a30 con 0x55b1cf947e30 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 -- 172.21.15.16:0/3877582954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 msgr2=0x55b1cf94c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf94c930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7b5809b080 tx=0x7f7b5809aec0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 -- 172.21.15.16:0/3877582954 shutdown_connections 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf94f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf94c930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 --2- 172.21.15.16:0/3877582954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947440 0x55b1cf947860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.662+0000 7f7b70b79740 1 -- 172.21.15.16:0/3877582954 >> 172.21.15.16:0/3877582954 conn(0x55b1cf942ab0 msgr2=0x55b1cf942eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 -- 172.21.15.16:0/3877582954 shutdown_connections 2024-04-25T01:26:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 -- 172.21.15.16:0/3877582954 wait complete. 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 -- start start 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 Processor -- start 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947440 0x55b1cf951800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf951d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf958d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf94fef0 con 0x55b1cf947e30 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf866cf0 con 0x55b1cf94cfa0 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf950070 con 0x55b1cf947440 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b70b79740 10 client.0 _ll_register_callbacks cb 0x55b1cf8666e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf951d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b66ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf951d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58228/0 (socket says 172.21.15.16:58228) 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.663+0000 7f7b66ffd640 1 -- 172.21.15.16:0/441208150 learned_addr learned my addr 172.21.15.16:0/441208150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b67fff640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf958d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b677fe640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947440 0x55b1cf951800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b66ffd640 1 -- 172.21.15.16:0/441208150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947440 msgr2=0x55b1cf951800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947440 0x55b1cf951800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b66ffd640 1 -- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 msgr2=0x55b1cf958d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf958d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b66ffd640 1 -- 172.21.15.16:0/441208150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1cf6896b0 con 0x55b1cf947e30 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf951d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0314e0 tx=0x7f7b5c004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8a649ce57f366d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b46ffd640 1 -- 172.21.15.16:0/441208150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c035040 con 0x55b1cf947e30 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b46ffd640 1 -- 172.21.15.16:0/441208150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7b5c003ac0 con 0x55b1cf947e30 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5809ba80 con 0x55b1cf947e30 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b46ffd640 1 -- 172.21.15.16:0/441208150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b5c04c070 con 0x55b1cf947e30 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b70b79740 10 client.15084 fetch_fsmap learned FSMap version 21 2024-04-25T01:26:48.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b1cf95bdd0 con 0x55b1cf947e30 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b46ffd640 1 -- 172.21.15.16:0/441208150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b5809ba80 con 0x55b1cf947e30 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.664+0000 7f7b46ffd640 1 -- 172.21.15.16:0/441208150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7b5c03c440 con 0x55b1cf947e30 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b70b79740 10 client.15084 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b70b79740 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b70b79740 20 client.15084 populate_metadata read hostname 'smithi016' 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b70b79740 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf95e4e0 0x55b1cf95f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b70b79740 4 client.15084 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1cf8b8440 con 0x55b1cf95e4e0 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b677fe640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf95e4e0 0x55b1cf95f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b46ffd640 10 client.15084 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.665+0000 7f7b677fe640 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf95e4e0 0x55b1cf95f950 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b90a3c3db9df6c4 server_cookie=c0846edb6b05725 in_seq=0 out_seq=0 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.666+0000 7f7b46ffd640 1 -- 172.21.15.16:0/441208150 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1cf8b8440 con 0x55b1cf95e4e0 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.666+0000 7f7b46ffd640 10 client.15084 handle_command_reply: tid=0 2024-04-25T01:26:48.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.666+0000 7f7b70b79740 1 client.15084 shutdown 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.666+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 msgr2=0x55b1cf951d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.666+0000 7f7b70b79740 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf951d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0314e0 tx=0x7f7b5c004440 comp rx=0 tx=0).stop 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 shutdown_connections 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf95e4e0 0x55b1cf95f950 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf958d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947e30 0x55b1cf951d40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b46ffd640 0 client.15084 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 --2- 172.21.15.16:0/441208150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947440 0x55b1cf951800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 >> 172.21.15.16:0/441208150 conn(0x55b1cf942ab0 msgr2=0x55b1cf941ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 shutdown_connections 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 1 -- 172.21.15.16:0/441208150 wait complete. 2024-04-25T01:26:48.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.667+0000 7f7b70b79740 20 client.15084 trim_cache size 0 max 16384 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 Processor -- start 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 -- start start 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf94c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947e30 0x55b1cf958d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf941250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf95dc90 con 0x55b1cf947440 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf95e360 con 0x55b1cf94cfa0 2024-04-25T01:26:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.685+0000 7f7b70b79740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf95ea30 con 0x55b1cf947e30 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf94c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf94c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58236/0 (socket says 172.21.15.16:58236) 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 -- 172.21.15.16:0/2199379754 learned_addr learned my addr 172.21.15.16:0/2199379754 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 -- 172.21.15.16:0/2199379754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947e30 msgr2=0x55b1cf958d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b67fff640 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf941250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947e30 0x55b1cf958d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 -- 172.21.15.16:0/2199379754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 msgr2=0x55b1cf941250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf941250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 -- 172.21.15.16:0/2199379754 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1cf947270 con 0x55b1cf947440 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b677fe640 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf94c820 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7b58000c00 tx=0x7f7b5809aff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3889a6aac8146bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b4d7fa640 1 -- 172.21.15.16:0/2199379754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5809ba80 con 0x55b1cf947440 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b4d7fa640 1 -- 172.21.15.16:0/2199379754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b5809f730 con 0x55b1cf947440 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.686+0000 7f7b4d7fa640 1 -- 172.21.15.16:0/2199379754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58009c90 con 0x55b1cf947440 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- 172.21.15.16:0/2199379754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 msgr2=0x55b1cf94c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf94c820 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7b58000c00 tx=0x7f7b5809aff0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- 172.21.15.16:0/2199379754 shutdown_connections 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf94cfa0 0x55b1cf941250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf947e30 0x55b1cf958d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- 172.21.15.16:0/2199379754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf94c820 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- 172.21.15.16:0/2199379754 >> 172.21.15.16:0/2199379754 conn(0x55b1cf942ab0 msgr2=0x55b1cf9486c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- 172.21.15.16:0/2199379754 shutdown_connections 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- 172.21.15.16:0/2199379754 wait complete. 2024-04-25T01:26:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- start start 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 Processor -- start 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf956f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947e30 0x55b1cf9574c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf957af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf95f7d0 con 0x55b1cf947440 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf95f100 con 0x55b1cf947e30 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.687+0000 7f7b70b79740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b1cf95fea0 con 0x55b1cf94cfa0 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b70b79740 10 client.0 _ll_register_callbacks cb 0x7f7b5c0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf956f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf956f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58242/0 (socket says 172.21.15.16:58242) 2024-04-25T01:26:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 -- 172.21.15.16:0/250763793 learned_addr learned my addr 172.21.15.16:0/250763793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947e30 0x55b1cf9574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 -- 172.21.15.16:0/250763793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 msgr2=0x55b1cf957af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf957af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 -- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947e30 msgr2=0x55b1cf9574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947e30 0x55b1cf9574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 -- 172.21.15.16:0/250763793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1cf958320 con 0x55b1cf947440 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b677fe640 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf956f80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7b580bafd0 tx=0x7f7b5809b1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e1d1e915b29b27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b46ffd640 1 -- 172.21.15.16:0/250763793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5809a9d0 con 0x55b1cf947440 2024-04-25T01:26:48.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b46ffd640 1 -- 172.21.15.16:0/250763793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7b580b4110 con 0x55b1cf947440 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5809f730 con 0x55b1cf947440 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b46ffd640 1 -- 172.21.15.16:0/250763793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b580b4350 con 0x55b1cf947440 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b70b79740 10 client.15090 fetch_fsmap learned FSMap version 21 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b1cf9c47f0 con 0x55b1cf947440 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.688+0000 7f7b46ffd640 1 -- 172.21.15.16:0/250763793 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b5809f730 con 0x55b1cf947440 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b46ffd640 1 -- 172.21.15.16:0/250763793 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7b580970d0 con 0x55b1cf947440 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b70b79740 10 client.15090 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b70b79740 10 client.15090 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:26:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b70b79740 20 client.15090 populate_metadata read hostname 'smithi016' 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b70b79740 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf9c8ac0 0x55b1cf9c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b70b79740 4 client.15090 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "status"}) -- 0x55b1cf8b2790 con 0x55b1cf9c8ac0 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf9c8ac0 0x55b1cf9c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b46ffd640 10 client.15090 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b66ffd640 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf9c8ac0 0x55b1cf9c77d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f407c62084bc447 server_cookie=bfe580ffe95d1c07 in_seq=0 out_seq=0 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.689+0000 7f7b46ffd640 1 -- 172.21.15.16:0/250763793 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b1cf8b2790 con 0x55b1cf9c8ac0 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b46ffd640 10 client.15090 handle_command_reply: tid=0 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 client.15090 shutdown 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 msgr2=0x55b1cf956f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf956f80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7b580bafd0 tx=0x7f7b5809b1d0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 shutdown_connections 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1cf94cfa0 0x55b1cf957af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b1cf9c8ac0 0x55b1cf9c77d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1cf947e30 0x55b1cf9574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 --2- 172.21.15.16:0/250763793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b1cf947440 0x55b1cf956f80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 >> 172.21.15.16:0/250763793 conn(0x55b1cf942ab0 msgr2=0x55b1cf94c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b46ffd640 0 client.15090 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 shutdown_connections 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 1 -- 172.21.15.16:0/250763793 wait complete. 2024-04-25T01:26:48.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b70b79740 20 client.15090 trim_cache size 0 max 16384 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24524, 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 91.649202879000001, 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 21, 2024-04-25T01:26:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 87, 2024-04-25T01:26:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-04-25T01:26:48.680 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 97.919425938000003 2024-04-25T01:26:48.680 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:26:48.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7b500a2c10 msgr2=0x7f7b500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7b500a2c10 0x7f7b500a50d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55b1cf948690 tx=0x7f7b60009500 comp rx=0 tx=0).stop 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 msgr2=0x7f7b68101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 0x7f7b68101b90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800d950 tx=0x7f7b5800de10 comp rx=0 tx=0).stop 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 shutdown_connections 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7b500a2c10 0x7f7b500a50d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68107480 0x7f7b68100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b68104b50 0x7f7b681001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 --2- 172.21.15.16:0/2419859729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b68102220 0x7f7b68101b90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:48.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.690+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 >> 172.21.15.16:0/2419859729 conn(0x7f7b680fbc10 msgr2=0x7f7b68102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.691+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 shutdown_connections 2024-04-25T01:26:48.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:48.691+0000 7f7b6de6b640 1 -- 172.21.15.16:0/2419859729 wait complete. 2024-04-25T01:26:48.683 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T01:26:48.684 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:26:48.693 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:26:48.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T01:26:48.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:48 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2552868221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:48 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2324169473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:48 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/276456551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:48 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2552868221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:48 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2324169473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:48 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/276456551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:26:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:49.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:49 smithi016 ceph-mon[29153]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:49.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:49 smithi038 ceph-mon[32841]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:49.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:49 smithi156 ceph-mon[32631]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:50.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 -- 172.21.15.16:0/2670106540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 msgr2=0x7f8e54102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:50.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/2670106540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54102830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8e440099c0 tx=0x7f8e44009cd0 comp rx=0 tx=0).stop 2024-04-25T01:26:50.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 -- 172.21.15.16:0/2670106540 shutdown_connections 2024-04-25T01:26:50.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/2670106540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 0x7f8e54109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/2670106540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e541011d0 0x7f8e541015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/2670106540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54102830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 -- 172.21.15.16:0/2670106540 >> 172.21.15.16:0/2670106540 conn(0x7f8e540fc790 msgr2=0x7f8e540febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 -- 172.21.15.16:0/2670106540 shutdown_connections 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.835+0000 7f8e58d9f700 1 -- 172.21.15.16:0/2670106540 wait complete. 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 Processor -- start 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 -- start start 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e541011d0 0x7f8e54197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 0x7f8e5419e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410cd20 con 0x7f8e541023d0 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410cba0 con 0x7f8e54102d70 2024-04-25T01:26:50.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e58d9f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410cea0 con 0x7f8e541011d0 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e51d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e51d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58266/0 (socket says 172.21.15.16:58266) 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.836+0000 7f8e51d9b700 1 -- 172.21.15.16:0/1254760389 learned_addr learned my addr 172.21.15.16:0/1254760389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e5259c700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e541011d0 0x7f8e54197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e52d9d700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 0x7f8e5419e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e51d9b700 1 -- 172.21.15.16:0/1254760389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e541011d0 msgr2=0x7f8e54197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e51d9b700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e541011d0 0x7f8e54197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e51d9b700 1 -- 172.21.15.16:0/1254760389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 msgr2=0x7f8e5419e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e51d9b700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 0x7f8e5419e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e51d9b700 1 -- 172.21.15.16:0/1254760389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5419ef60 con 0x7f8e541023d0 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e52d9d700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 0x7f8e5419e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e51d9b700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54197840 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400b580 tx=0x7f8e44003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:50.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44027070 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e44004210 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5419f1f0 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400f460 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.837+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5419f720 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.838+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e54068f20 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.839+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8e4402a070 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.840+0000 7f8e437fe700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8e3c0a2ca0 0x7f8e3c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.840+0000 7f8e5259c700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8e3c0a2ca0 0x7f8e3c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.840+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8e440d09a0 con 0x7f8e541023d0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.840+0000 7f8e5259c700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8e3c0a2ca0 0x7f8e3c0a5160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8e48005fd0 tx=0x7f8e48005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:50.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:50.844+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e44031020 con 0x7f8e541023d0 2024-04-25T01:26:51.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.065+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f8e54062730 con 0x7f8e3c0a2ca0 2024-04-25T01:26:51.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.067+0000 7f8e437fe700 1 -- 172.21.15.16:0/1254760389 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f8e54062730 con 0x7f8e3c0a2ca0 2024-04-25T01:26:51.046 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-04-25T01:26:51.046 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-04-25T01:26:51.046 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-04-25T01:26:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8e3c0a2ca0 msgr2=0x7f8e3c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8e3c0a2ca0 0x7f8e3c0a5160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8e48005fd0 tx=0x7f8e48005e50 comp rx=0 tx=0).stop 2024-04-25T01:26:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 msgr2=0x7f8e54197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54197840 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400b580 tx=0x7f8e44003680 comp rx=0 tx=0).stop 2024-04-25T01:26:51.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 shutdown_connections 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8e3c0a2ca0 0x7f8e3c0a5160 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e54102d70 0x7f8e5419e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e541011d0 0x7f8e54197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 --2- 172.21.15.16:0/1254760389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e541023d0 0x7f8e54197840 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 >> 172.21.15.16:0/1254760389 conn(0x7f8e540fc790 msgr2=0x7f8e541056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 shutdown_connections 2024-04-25T01:26:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:51.069+0000 7f8e58d9f700 1 -- 172.21.15.16:0/1254760389 wait complete. 2024-04-25T01:26:51.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:51 smithi016 ceph-mon[29153]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:51.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:51 smithi038 ceph-mon[32841]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:51.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T01:26:51.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:51 smithi156 ceph-mon[32631]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:52.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:52 smithi038 ceph-mon[32841]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:52.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:52 smithi156 ceph-mon[32631]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:52.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:52 smithi016 ceph-mon[29153]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:53.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:53 smithi016 ceph-mon[29153]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:53.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:53 smithi038 ceph-mon[32841]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:53.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:53 smithi156 ceph-mon[32631]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:26:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 -- 172.21.15.16:0/1274774523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8101180 msgr2=0x7f8ec81015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/1274774523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8101180 0x7f8ec81015a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000b0d0 tx=0x7f8ec000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:26:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 -- 172.21.15.16:0/1274774523 shutdown_connections 2024-04-25T01:26:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/1274774523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ec8102d20 0x7f8ec81095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/1274774523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8101180 0x7f8ec81015a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:53.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/1274774523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ec8102380 0x7f8ec81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:53.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 -- 172.21.15.16:0/1274774523 >> 172.21.15.16:0/1274774523 conn(0x7f8ec80fc760 msgr2=0x7f8ec80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:53.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 -- 172.21.15.16:0/1274774523 shutdown_connections 2024-04-25T01:26:53.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 -- 172.21.15.16:0/1274774523 wait complete. 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 Processor -- start 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.790+0000 7f8ed07f7700 1 -- start start 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ed07f7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ec8101180 0x7f8ec81050c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ed07f7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ec8102380 0x7f8ec8103760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ed07f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 0x7f8ec8103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ed07f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec810cc60 con 0x7f8ec8102d20 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ed07f7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec810cae0 con 0x7f8ec8101180 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ed07f7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec810cde0 con 0x7f8ec8102380 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 0x7f8ec8103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 0x7f8ec8103ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39532/0 (socket says 172.21.15.16:39532) 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 -- 172.21.15.16:0/3481890428 learned_addr learned my addr 172.21.15.16:0/3481890428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ecdd92700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ec8102380 0x7f8ec8103760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ece593700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ec8101180 0x7f8ec81050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 -- 172.21.15.16:0/3481890428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ec8102380 msgr2=0x7f8ec8103760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ec8102380 0x7f8ec8103760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 -- 172.21.15.16:0/3481890428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ec8101180 msgr2=0x7f8ec81050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ec8101180 0x7f8ec81050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 -- 172.21.15.16:0/3481890428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec8104570 con 0x7f8ec8102d20 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8eced94700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 0x7f8ec8103ca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800c970 tx=0x7f8eb800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8007b00 con 0x7f8ec8102d20 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8eb8004ce0 con 0x7f8ec8102d20 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.791+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8005590 con 0x7f8ec8102d20 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.792+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec81a2560 con 0x7f8ec8102d20 2024-04-25T01:26:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.792+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec81a2a20 con 0x7f8ec8102d20 2024-04-25T01:26:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.792+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec8068f20 con 0x7f8ec8102d20 2024-04-25T01:26:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.794+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8eb8004750 con 0x7f8ec8102d20 2024-04-25T01:26:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.794+0000 7f8ec77fe700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8eb00a2ca0 0x7f8eb00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.794+0000 7f8ece593700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8eb00a2ca0 0x7f8eb00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.794+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8eb80cf000 con 0x7f8ec8102d20 2024-04-25T01:26:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.795+0000 7f8ece593700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8eb00a2ca0 0x7f8eb00a5160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000f80 tx=0x7f8ec000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:53.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:53.798+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8eb8014030 con 0x7f8ec8102d20 2024-04-25T01:26:53.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.013+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f8ec80fe950 con 0x7f8eb00a2ca0 2024-04-25T01:26:54.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.027+0000 7f8ec77fe700 1 -- 172.21.15.16:0/3481890428 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f8ec80fe950 con 0x7f8eb00a2ca0 2024-04-25T01:26:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (106s) 17s ago 106s 13.8M - 19.0.0-2445-gd2c45d97 f68e37b807ef 318e2f554f04 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi038 running (103s) 36s ago 103s 26.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef 305e466c097b 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi156 running (118s) 37s ago 117s 17.9M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9efd5d18a612 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (2m) 17s ago 2m 16.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef 30acd667c779 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi038 running (2m) 36s ago 2m 19.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef 6ffe41c2eb68 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi156 running (2m) 37s ago 2m 19.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef 554da208df0b 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (109s) 17s ago 109s 12.8M - 19.0.0-2445-gd2c45d97 f68e37b807ef b451c0c1eadf 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi038 running (115s) 36s ago 114s 19.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef 0bc659dae5f4 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi156 running (112s) 37s ago 112s 17.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 492b7fcb72ca 2024-04-25T01:26:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (2m) 17s ago 2m 13.8M - 19.0.0-2445-gd2c45d97 f68e37b807ef 8d26326a4cc6 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi038 running (2m) 36s ago 2m 15.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef b423c9ba0b88 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi156 running (2m) 37s ago 2m 16.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4cf67c1f405b 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (9m) 17s ago 9m 516M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4c4a282d44c9 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi038 *:8443,8765 running (7m) 36s ago 7m 452M - 19.0.0-2445-gd2c45d97 f68e37b807ef 68bae53df6de 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi156 *:8443,8765 running (7m) 37s ago 7m 452M - 19.0.0-2445-gd2c45d97 f68e37b807ef ff8816147300 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (9m) 17s ago 9m 49.6M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 80597ad1e310 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi038 running (7m) 36s ago 7m 41.8M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 97c40aa9d481 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi156 running (8m) 37s ago 8m 38.2M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef d84995c16b88 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 17s ago 7m 55.5M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef db916fc9e14e 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi038 running (6m) 36s ago 6m 48.0M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 5e6891697200 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi156 running (6m) 37s ago 6m 56.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 15408ca9b32a 2024-04-25T01:26:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 17s ago 6m 42.6M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 58477582aa77 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi038 running (5m) 36s ago 5m 45.4M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 4a84b8775b71 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi156 running (5m) 37s ago 5m 46.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 245c957b9d59 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (4m) 17s ago 4m 40.9M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 744b86c5ff2e 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi038 running (4m) 36s ago 4m 46.0M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef da18fb9c9517 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi156 running (3m) 37s ago 3m 44.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 961a9a329c8a 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (3m) 17s ago 3m 53.7M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef ab6658352a4f 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi038 running (3m) 36s ago 3m 43.9M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 4ee1970d7f7a 2024-04-25T01:26:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi156 running (2m) 37s ago 2m 52.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 15bf8474b8f9 2024-04-25T01:26:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8eb00a2ca0 msgr2=0x7f8eb00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8eb00a2ca0 0x7f8eb00a5160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000f80 tx=0x7f8ec000b040 comp rx=0 tx=0).stop 2024-04-25T01:26:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 msgr2=0x7f8ec8103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 0x7f8ec8103ca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800c970 tx=0x7f8eb800cd30 comp rx=0 tx=0).stop 2024-04-25T01:26:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 shutdown_connections 2024-04-25T01:26:54.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8eb00a2ca0 0x7f8eb00a5160 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ec8102d20 0x7f8ec8103ca0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ec8101180 0x7f8ec81050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 --2- 172.21.15.16:0/3481890428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ec8102380 0x7f8ec8103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 >> 172.21.15.16:0/3481890428 conn(0x7f8ec80fc760 msgr2=0x7f8ec80fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 shutdown_connections 2024-04-25T01:26:54.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:54.029+0000 7f8ed07f7700 1 -- 172.21.15.16:0/3481890428 wait complete. 2024-04-25T01:26:54.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T01:26:54.948 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:55.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:55 smithi016 ceph-mon[29153]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:55.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:55 smithi016 ceph-mon[29153]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:55.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:55 smithi038 ceph-mon[32841]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:55.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:55 smithi038 ceph-mon[32841]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:55.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:55 smithi156 ceph-mon[32631]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:26:55.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:55 smithi156 ceph-mon[32631]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/1501221600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 msgr2=0x7f5fdc103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/1501221600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc103820 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80099c0 tx=0x7f5fc8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/1501221600 shutdown_connections 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/1501221600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fdc103d60 0x7f5fdc10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/1501221600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fdc1021c0 0x7f5fdc1025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/1501221600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc103820 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/1501221600 >> 172.21.15.16:0/1501221600 conn(0x7f5fdc0fd760 msgr2=0x7f5fdc0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/1501221600 shutdown_connections 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.868+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/1501221600 wait complete. 2024-04-25T01:26:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 Processor -- start 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 -- start start 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fdc1021c0 0x7f5fdc198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fdc103d60 0x7f5fdc19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc10dc80 con 0x7f5fdc1033c0 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc10db00 con 0x7f5fdc103d60 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe1f9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc10de00 con 0x7f5fdc1021c0 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fdbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fdbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39546/0 (socket says 172.21.15.16:39546) 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fdbfff700 1 -- 172.21.15.16:0/473544546 learned_addr learned my addr 172.21.15.16:0/473544546 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe179c700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fdc103d60 0x7f5fdc19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.869+0000 7f5fe0f9b700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fdc1021c0 0x7f5fdc198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fdbfff700 1 -- 172.21.15.16:0/473544546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fdc1021c0 msgr2=0x7f5fdc198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fdbfff700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fdc1021c0 0x7f5fdc198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fdbfff700 1 -- 172.21.15.16:0/473544546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fdc103d60 msgr2=0x7f5fdc19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fdbfff700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fdc103d60 0x7f5fdc19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fdbfff700 1 -- 172.21.15.16:0/473544546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc19fe90 con 0x7f5fdc1033c0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fdbfff700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc198770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800b580 tx=0x7f5fc80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8027070 con 0x7f5fdc1033c0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc8004110 con 0x7f5fdc1033c0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fdc1a0120 con 0x7f5fdc1033c0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8022da0 con 0x7f5fdc1033c0 2024-04-25T01:26:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.870+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fdc1a05d0 con 0x7f5fdc1033c0 2024-04-25T01:26:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.871+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fdc068f20 con 0x7f5fdc1033c0 2024-04-25T01:26:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.872+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5fc802a070 con 0x7f5fdc1033c0 2024-04-25T01:26:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.873+0000 7f5fd9ffb700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f5fcc0a2ca0 0x7f5fcc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.873+0000 7f5fe0f9b700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f5fcc0a2ca0 0x7f5fcc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.873+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5fc80d0870 con 0x7f5fdc1033c0 2024-04-25T01:26:56.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.874+0000 7f5fe0f9b700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f5fcc0a2ca0 0x7f5fcc0a5160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc103220 tx=0x7f5fd000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:56.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:56.877+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5fc8031020 con 0x7f5fdc1033c0 2024-04-25T01:26:57.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.091+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f5fdc062670 con 0x7f5fcc0a2ca0 2024-04-25T01:26:57.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.098+0000 7f5fd9ffb700 1 -- 172.21.15.16:0/473544546 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f5fdc062670 con 0x7f5fcc0a2ca0 2024-04-25T01:26:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T01:26:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 40s ago 2m smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T01:26:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 40s ago 7m smithi016=x;smithi038=y;smithi156=z;count:3 2024-04-25T01:26:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 40s ago 8m smithi016:172.21.15.16=a;smithi038:172.21.15.38=b;smithi156:172.21.15.156=c;count:3 2024-04-25T01:26:57.077 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 40s ago - 2024-04-25T01:26:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f5fcc0a2ca0 msgr2=0x7f5fcc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f5fcc0a2ca0 0x7f5fcc0a5160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc103220 tx=0x7f5fd000a380 comp rx=0 tx=0).stop 2024-04-25T01:26:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 msgr2=0x7f5fdc198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc198770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800b580 tx=0x7f5fc80057c0 comp rx=0 tx=0).stop 2024-04-25T01:26:57.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 shutdown_connections 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f5fcc0a2ca0 0x7f5fcc0a5160 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fdc103d60 0x7f5fdc19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fdc1021c0 0x7f5fdc198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 --2- 172.21.15.16:0/473544546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fdc1033c0 0x7f5fdc198770 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 >> 172.21.15.16:0/473544546 conn(0x7f5fdc0fd760 msgr2=0x7f5fdc1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 shutdown_connections 2024-04-25T01:26:57.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:57.100+0000 7f5fe1f9d700 1 -- 172.21.15.16:0/473544546 wait complete. 2024-04-25T01:26:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:57 smithi038 ceph-mon[32841]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:26:57.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:57 smithi156 ceph-mon[32631]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:26:57.835 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:57 smithi016 ceph-mon[29153]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:26:57.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T01:26:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:26:58.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:58 smithi016 ceph-mon[29153]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:58.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:58 smithi038 ceph-mon[32841]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:58.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:58 smithi156 ceph-mon[32631]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:59.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:26:59 smithi016 ceph-mon[29153]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:59.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:26:59 smithi038 ceph-mon[32841]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:26:59.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:26:59 smithi156 ceph-mon[32631]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 -- 172.21.15.16:0/3768556475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44101190 msgr2=0x7fda441015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 --2- 172.21.15.16:0/3768556475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44101190 0x7fda441015b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fda400099c0 tx=0x7fda40009cd0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 -- 172.21.15.16:0/3768556475 shutdown_connections 2024-04-25T01:27:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 --2- 172.21.15.16:0/3768556475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 0x7fda441095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 --2- 172.21.15.16:0/3768556475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44101190 0x7fda441015b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 --2- 172.21.15.16:0/3768556475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda44102390 0x7fda441027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 -- 172.21.15.16:0/3768556475 >> 172.21.15.16:0/3768556475 conn(0x7fda440fc730 msgr2=0x7fda440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 -- 172.21.15.16:0/3768556475 shutdown_connections 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.959+0000 7fda4c064700 1 -- 172.21.15.16:0/3768556475 wait complete. 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 Processor -- start 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 -- start start 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda44101190 0x7fda44198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 0x7fda44198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 0x7fda4419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda4410cd00 con 0x7fda44102390 2024-04-25T01:27:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fda4410cb80 con 0x7fda44102d30 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4c064700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda4410ce80 con 0x7fda44101190 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 0x7fda44198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 0x7fda44198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39566/0 (socket says 172.21.15.16:39566) 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 -- 172.21.15.16:0/1984444909 learned_addr learned my addr 172.21.15.16:0/1984444909 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda49e00700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda44101190 0x7fda44198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 -- 172.21.15.16:0/1984444909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda44101190 msgr2=0x7fda44198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda44101190 0x7fda44198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 -- 172.21.15.16:0/1984444909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 msgr2=0x7fda4419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda4a601700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 0x7fda4419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.960+0000 7fda495ff700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 0x7fda4419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda495ff700 1 -- 172.21.15.16:0/1984444909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda441a01a0 con 0x7fda44102390 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda4a601700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 0x7fda4419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda495ff700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 0x7fda44198a80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b560 tx=0x7fda3400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34015020 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda441a0490 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda34011ce0 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34010da0 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.961+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda441a0a50 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.962+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda44068f20 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.963+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fda34018020 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.963+0000 7fda3affd700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fda300a2c20 0x7fda300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.964+0000 7fda49e00700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fda300a2c20 0x7fda300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.964+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fda340ce960 con 0x7fda44102390 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.964+0000 7fda49e00700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fda300a2c20 0x7fda300a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fda40000c00 tx=0x7fda40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:26:59.968+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fda3401f020 con 0x7fda44102390 2024-04-25T01:27:00.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.183+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fda441a0d70 con 0x7fda300a2c20 2024-04-25T01:27:00.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.184+0000 7fda3affd700 1 -- 172.21.15.16:0/1984444909 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fda441a0d70 con 0x7fda300a2c20 2024-04-25T01:27:00.164 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-04-25T01:27:00.164 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-04-25T01:27:00.164 INFO:teuthology.orchestra.run.smithi016.stdout:smithi038 172.21.15.38 2024-04-25T01:27:00.164 INFO:teuthology.orchestra.run.smithi016.stdout:smithi156 172.21.15.156 2024-04-25T01:27:00.164 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fda300a2c20 msgr2=0x7fda300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fda300a2c20 0x7fda300a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fda40000c00 tx=0x7fda40005fd0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 msgr2=0x7fda44198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 0x7fda44198a80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b560 tx=0x7fda3400b920 comp rx=0 tx=0).stop 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 shutdown_connections 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fda300a2c20 0x7fda300a50e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda44102d30 0x7fda4419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda44101190 0x7fda44198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 --2- 172.21.15.16:0/1984444909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda44102390 0x7fda44198a80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.186+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 >> 172.21.15.16:0/1984444909 conn(0x7fda440fc730 msgr2=0x7fda44069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:00.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.187+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 shutdown_connections 2024-04-25T01:27:00.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:00.187+0000 7fda4c064700 1 -- 172.21.15.16:0/1984444909 wait complete. 2024-04-25T01:27:01.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T01:27:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:27:01.635 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:01 smithi016 ceph-mon[29153]: pgmap v347: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:01.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:01 smithi016 ceph-mon[29153]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:01 smithi038 ceph-mon[32841]: pgmap v347: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:01 smithi038 ceph-mon[32841]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:01.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:01 smithi156 ceph-mon[32631]: pgmap v347: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:01.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:01 smithi156 ceph-mon[32631]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 -- 172.21.15.16:0/2070165174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341034b0 msgr2=0x7ff434103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 --2- 172.21.15.16:0/2070165174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341034b0 0x7ff434103910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff4200099c0 tx=0x7ff420009cd0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 -- 172.21.15.16:0/2070165174 shutdown_connections 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 --2- 172.21.15.16:0/2070165174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff434103e50 0x7ff43410a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 --2- 172.21.15.16:0/2070165174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341022b0 0x7ff4341026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 --2- 172.21.15.16:0/2070165174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341034b0 0x7ff434103910 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 -- 172.21.15.16:0/2070165174 >> 172.21.15.16:0/2070165174 conn(0x7ff4340fd850 msgr2=0x7ff4340ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 -- 172.21.15.16:0/2070165174 shutdown_connections 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 -- 172.21.15.16:0/2070165174 wait complete. 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.075+0000 7ff43aa68700 1 Processor -- start 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 -- start start 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 0x7ff4341983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341034b0 0x7ff4341988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff434103e50 0x7ff43419f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff43410dd40 con 0x7ff4341022b0 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff43410dbc0 con 0x7ff4341034b0 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff43aa68700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff43410dec0 con 0x7ff434103e50 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 0x7ff4341983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 0x7ff4341983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41716/0 (socket says 172.21.15.16:41716) 2024-04-25T01:27:03.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 -- 172.21.15.16:0/3752414236 learned_addr learned my addr 172.21.15.16:0/3752414236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff433fff700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341034b0 0x7ff4341988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 -- 172.21.15.16:0/3752414236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff434103e50 msgr2=0x7ff43419f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff434103e50 0x7ff43419f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 -- 172.21.15.16:0/3752414236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341034b0 msgr2=0x7ff4341988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341034b0 0x7ff4341988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 -- 172.21.15.16:0/3752414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4341a0010 con 0x7ff4341022b0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff433fff700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341034b0 0x7ff4341988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff438804700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 0x7ff4341983b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c9a0 tx=0x7ff42800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.076+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428004220 con 0x7ff4341022b0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.077+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff428007d10 con 0x7ff4341022b0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.077+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4341a0300 con 0x7ff4341022b0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.077+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428003e10 con 0x7ff4341022b0 2024-04-25T01:27:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.077+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a0890 con 0x7ff4341022b0 2024-04-25T01:27:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.078+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff434068f20 con 0x7ff4341022b0 2024-04-25T01:27:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.079+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff428007860 con 0x7ff4341022b0 2024-04-25T01:27:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.080+0000 7ff431ffb700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4240a2ca0 0x7ff4240a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.080+0000 7ff433fff700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4240a2ca0 0x7ff4240a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:03.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.080+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff4280cec50 con 0x7ff4341022b0 2024-04-25T01:27:03.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.080+0000 7ff433fff700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4240a2ca0 0x7ff4240a5160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff420006010 tx=0x7ff420005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.084+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4280d4020 con 0x7ff4341022b0 2024-04-25T01:27:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.299+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff434062670 con 0x7ff4240a2ca0 2024-04-25T01:27:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.301+0000 7ff431ffb700 1 -- 172.21.15.16:0/3752414236 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7ff434062670 con 0x7ff4240a2ca0 2024-04-25T01:27:03.280 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T01:27:03.280 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:27:03.280 INFO:teuthology.orchestra.run.smithi016.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:27:03.280 INFO:teuthology.orchestra.run.smithi016.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-25T01:27:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4240a2ca0 msgr2=0x7ff4240a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4240a2ca0 0x7ff4240a5160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff420006010 tx=0x7ff420005c40 comp rx=0 tx=0).stop 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 msgr2=0x7ff4341983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 0x7ff4341983b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c9a0 tx=0x7ff42800cd60 comp rx=0 tx=0).stop 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 shutdown_connections 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7ff4240a2ca0 0x7ff4240a5160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff434103e50 0x7ff43419f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4341022b0 0x7ff4341983b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 --2- 172.21.15.16:0/3752414236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4341034b0 0x7ff4341988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 >> 172.21.15.16:0/3752414236 conn(0x7ff4340fd850 msgr2=0x7ff4341067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 shutdown_connections 2024-04-25T01:27:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:03.303+0000 7ff43aa68700 1 -- 172.21.15.16:0/3752414236 wait complete. 2024-04-25T01:27:03.578 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:03 smithi016 ceph-mon[29153]: pgmap v348: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:27:03.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:03 smithi038 ceph-mon[32841]: pgmap v348: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:27:03.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:03 smithi156 ceph-mon[32631]: pgmap v348: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:27:04.009 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:27:04.018 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:27:04.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T01:27:04.270 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:27:04.660 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:04 smithi016 ceph-mon[29153]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:04.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:04 smithi038 ceph-mon[32841]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:04.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:04 smithi156 ceph-mon[32631]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:05.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:05 smithi038 ceph-mon[32841]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:05.854 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:05 smithi156 ceph-mon[32631]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:05 smithi016 ceph-mon[29153]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 -- 172.21.15.16:0/292985837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f4103e50 msgr2=0x7fd8f410a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 --2- 172.21.15.16:0/292985837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f4103e50 0x7fd8f410a700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00b0d0 tx=0x7fd8ec00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 -- 172.21.15.16:0/292985837 shutdown_connections 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 --2- 172.21.15.16:0/292985837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f4103e50 0x7fd8f410a700 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 --2- 172.21.15.16:0/292985837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8f41022b0 0x7fd8f41026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 --2- 172.21.15.16:0/292985837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8f41034b0 0x7fd8f4103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 -- 172.21.15.16:0/292985837 >> 172.21.15.16:0/292985837 conn(0x7fd8f40fd850 msgr2=0x7fd8f40ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 -- 172.21.15.16:0/292985837 shutdown_connections 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.121+0000 7fd8fa527700 1 -- 172.21.15.16:0/292985837 wait complete. 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 Processor -- start 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 -- start start 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8f41022b0 0x7fd8f41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 0x7fd8f4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8f4103e50 0x7fd8f419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f410de40 con 0x7fd8f41034b0 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f410dcc0 con 0x7fd8f41022b0 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8fa527700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f410dfc0 con 0x7fd8f4103e50 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8f37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 0x7fd8f4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8f37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 0x7fd8f4198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41732/0 (socket says 172.21.15.16:41732) 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8f37fe700 1 -- 172.21.15.16:0/2262176518 learned_addr learned my addr 172.21.15.16:0/2262176518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8f8ac4700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8f4103e50 0x7fd8f419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.122+0000 7fd8f3fff700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8f41022b0 0x7fd8f41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f37fe700 1 -- 172.21.15.16:0/2262176518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8f4103e50 msgr2=0x7fd8f419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f37fe700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8f4103e50 0x7fd8f419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f37fe700 1 -- 172.21.15.16:0/2262176518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8f41022b0 msgr2=0x7fd8f41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f37fe700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8f41022b0 0x7fd8f41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f37fe700 1 -- 172.21.15.16:0/2262176518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f41a0130 con 0x7fd8f41034b0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f37fe700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 0x7fd8f4198a10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400b560 tx=0x7fd8e400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4015020 con 0x7fd8f41034b0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e4011ce0 con 0x7fd8f41034b0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f41a0420 con 0x7fd8f41034b0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4010d60 con 0x7fd8f41034b0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.123+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f41a09e0 con 0x7fd8f41034b0 2024-04-25T01:27:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.124+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f4068f20 con 0x7fd8f41034b0 2024-04-25T01:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.125+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd8e4018020 con 0x7fd8f41034b0 2024-04-25T01:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.126+0000 7fd8f17fa700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd8e00a2c20 0x7fd8e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.126+0000 7fd8f3fff700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd8e00a2c20 0x7fd8e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.126+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd8e40ce850 con 0x7fd8f41034b0 2024-04-25T01:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.126+0000 7fd8f3fff700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd8e00a2c20 0x7fd8e00a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4103310 tx=0x7fd8dc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.131+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8e401f020 con 0x7fd8f41034b0 2024-04-25T01:27:06.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.380+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fd8f4062e40 con 0x7fd8f41034b0 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.380+0000 7fd8f17fa700 1 -- 172.21.15.16:0/2262176518 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3046 (secure 0 0 0) 0x7fd8e409b790 con 0x7fd8f41034b0 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:e21 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 21 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T01:27:06.360 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-25T01:25:16.327985+0000 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-25T01:25:21.068325+0000 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 3 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2 2024-04-25T01:27:06.361 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24524,1=24499,2=24476} 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3] 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24524 members: 24476,24499,24524 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24524} state up:active seq 3 addr [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{0:24475} state up:standby-replay seq 1 addr [v2:172.21.15.156:6836/2504948433,v1:172.21.15.156:6837/2504948433] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{1:24499} state up:active seq 6 addr [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.362 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{1:24457} state up:standby-replay seq 1 addr [v2:172.21.15.156:6834/2186765223,v1:172.21.15.156:6835/2186765223] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{2:24476} state up:active seq 7 addr [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{2:24449} state up:standby-replay seq 1 addr [v2:172.21.15.38:6834/2628472521,v1:172.21.15.38:6835/2628472521] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{-1:14532} state up:standby seq 1 addr [v2:172.21.15.16:6834/690842306,v1:172.21.15.16:6835/690842306] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{-1:14544} state up:standby seq 1 addr [v2:172.21.15.16:6836/2204481226,v1:172.21.15.16:6837/2204481226] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{-1:14610} state up:standby seq 1 addr [v2:172.21.15.16:6838/4015716085,v1:172.21.15.16:6839/4015716085] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{-1:14628} state up:standby seq 1 addr [v2:172.21.15.16:6840/3747963498,v1:172.21.15.16:6841/3747963498] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{-1:24406} state up:standby seq 1 addr [v2:172.21.15.156:6832/1629519202,v1:172.21.15.156:6833/1629519202] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.363 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{-1:24419} state up:standby seq 1 addr [v2:172.21.15.38:6832/3531176332,v1:172.21.15.38:6833/3531176332] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:27:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.382+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd8e00a2c20 msgr2=0x7fd8e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.382+0000 7fd8fa527700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd8e00a2c20 0x7fd8e00a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4103310 tx=0x7fd8dc009190 comp rx=0 tx=0).stop 2024-04-25T01:27:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.382+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 msgr2=0x7fd8f4198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:06.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.382+0000 7fd8fa527700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 0x7fd8f4198a10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400b560 tx=0x7fd8e400b870 comp rx=0 tx=0).stop 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 shutdown_connections 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd8e00a2c20 0x7fd8e00a50e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8f4103e50 0x7fd8f419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8f41022b0 0x7fd8f41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 --2- 172.21.15.16:0/2262176518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f41034b0 0x7fd8f4198a10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 >> 172.21.15.16:0/2262176518 conn(0x7fd8f40fd850 msgr2=0x7fd8f4106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 shutdown_connections 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:06.383+0000 7fd8fa527700 1 -- 172.21.15.16:0/2262176518 wait complete. 2024-04-25T01:27:06.365 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:27:06.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:06 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2262176518' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:27:06.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:06 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2262176518' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:27:06.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:06 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2262176518' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:27:07.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T01:27:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:27:07.695 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:07 smithi016 ceph-mon[29153]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:27:07.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:07 smithi038 ceph-mon[32841]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:27:07.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:07 smithi156 ceph-mon[32631]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 -- 172.21.15.16:0/3468012895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a0101b00 msgr2=0x7fe4a0101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/3468012895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a0101b00 0x7fe4a0101f60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe49c0099c0 tx=0x7fe49c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 -- 172.21.15.16:0/3468012895 shutdown_connections 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/3468012895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a01024a0 0x7fe4a0108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/3468012895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 0x7fe4a0100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/3468012895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a0101b00 0x7fe4a0101f60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 -- 172.21.15.16:0/3468012895 >> 172.21.15.16:0/3468012895 conn(0x7fe4a00fbec0 msgr2=0x7fe4a00fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 -- 172.21.15.16:0/3468012895 shutdown_connections 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.287+0000 7fe4a85bb700 1 -- 172.21.15.16:0/3468012895 wait complete. 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 Processor -- start 2024-04-25T01:27:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 -- start start 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 0x7fe4a0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 0x7fe4a0198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 0x7fe4a019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a010cca0 con 0x7fe4a01024a0 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a010cb20 con 0x7fe4a0101b00 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a85bb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a010ce20 con 0x7fe4a0100900 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a6b58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 0x7fe4a019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a5b56700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 0x7fe4a0198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a6b58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 0x7fe4a019fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41754/0 (socket says 172.21.15.16:41754) 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a5b56700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 0x7fe4a0198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:38410/0 (socket says 172.21.15.16:38410) 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a6b58700 1 -- 172.21.15.16:0/793222498 learned_addr learned my addr 172.21.15.16:0/793222498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.288+0000 7fe4a6357700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 0x7fe4a0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a5b56700 1 -- 172.21.15.16:0/793222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 msgr2=0x7fe4a0198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a5b56700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 0x7fe4a0198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a5b56700 1 -- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 msgr2=0x7fe4a019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a5b56700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 0x7fe4a019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a5b56700 1 -- 172.21.15.16:0/793222498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a01a00b0 con 0x7fe4a0101b00 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a6b58700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 0x7fe4a019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:27:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a6357700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 0x7fe4a0198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a5b56700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 0x7fe4a0198990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe49c0098e0 tx=0x7fe49c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c027070 con 0x7fe4a0101b00 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a01a0340 con 0x7fe4a0101b00 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe49c0058e0 con 0x7fe4a0101b00 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c022b50 con 0x7fe4a0101b00 2024-04-25T01:27:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.289+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a01a08f0 con 0x7fe4a0101b00 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.290+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a0068f20 con 0x7fe4a0101b00 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.292+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe49c02a070 con 0x7fe4a0101b00 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.292+0000 7fe4937fe700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe48c0a2c20 0x7fe48c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.292+0000 7fe4a6357700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe48c0a2c20 0x7fe48c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.292+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe49c0d0770 con 0x7fe4a0101b00 2024-04-25T01:27:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.293+0000 7fe4a6357700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe48c0a2c20 0x7fe48c0a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0101960 tx=0x7fe494009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.296+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe49c031070 con 0x7fe4a0101b00 2024-04-25T01:27:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.505+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fe4a01a0b50 con 0x7fe4a0101b00 2024-04-25T01:27:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.507+0000 7fe4937fe700 1 -- 172.21.15.16:0/793222498 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5328 (secure 0 0 0) 0x7fe49c013070 con 0x7fe4a0101b00 2024-04-25T01:27:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 87 2024-04-25T01:27:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:27:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-25T01:17:02.974878+0000 2024-04-25T01:27:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-25T01:26:01.175296+0000 2024-04-25T01:27:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 26 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/1017436960,v1:172.21.15.16:6803/1017436960] [v2:172.21.15.16:6804/1017436960,v1:172.21.15.16:6805/1017436960] exists,up 8b40650a-6f71-49c8-b04f-f76ed7b91b72 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6800/4146278985,v1:172.21.15.38:6801/4146278985] [v2:172.21.15.38:6802/4146278985,v1:172.21.15.38:6803/4146278985] exists,up 3d73a4e5-3402-40e8-a625-53872c981f89 2024-04-25T01:27:09.487 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6800/380124115,v1:172.21.15.156:6801/380124115] [v2:172.21.15.156:6802/380124115,v1:172.21.15.156:6803/380124115] exists,up ffa52ec5-7d59-482c-bf75-87a3b7f21447 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/3649873026,v1:172.21.15.16:6811/3649873026] [v2:172.21.15.16:6812/3649873026,v1:172.21.15.16:6813/3649873026] exists,up 2c945cb2-a2ff-473b-98a7-2e462b7c59b6 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6808/1109232541,v1:172.21.15.38:6809/1109232541] [v2:172.21.15.38:6810/1109232541,v1:172.21.15.38:6811/1109232541] exists,up 5e91cb88-1603-4183-b475-1ab7df79597a 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6808/3440527374,v1:172.21.15.156:6809/3440527374] [v2:172.21.15.156:6810/3440527374,v1:172.21.15.156:6811/3440527374] exists,up 451201e8-0fc6-4319-b7d9-9be4425d6384 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/2018477454,v1:172.21.15.16:6819/2018477454] [v2:172.21.15.16:6820/2018477454,v1:172.21.15.16:6821/2018477454] exists,up f933fcbe-f953-4fb0-8e2f-5eec380563bd 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 51 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6816/2653364257,v1:172.21.15.38:6817/2653364257] [v2:172.21.15.38:6818/2653364257,v1:172.21.15.38:6819/2653364257] exists,up da57b8de-edaf-497f-afdb-0d5648bf0c3c 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6816/1052118798,v1:172.21.15.156:6817/1052118798] [v2:172.21.15.156:6818/1052118798,v1:172.21.15.156:6819/1052118798] exists,up d6850fc5-5935-403c-bbe2-c2fc2e73d3cd 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/2042480684,v1:172.21.15.16:6827/2042480684] [v2:172.21.15.16:6828/2042480684,v1:172.21.15.16:6829/2042480684] exists,up d90517c5-56a1-45f6-adf5-6c2b28702bc2 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6824/3034934986,v1:172.21.15.38:6825/3034934986] [v2:172.21.15.38:6826/3034934986,v1:172.21.15.38:6827/3034934986] exists,up 3da7ce51-60f7-45d6-87d3-42baee7a62db 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6824/1283302638,v1:172.21.15.156:6825/1283302638] [v2:172.21.15.156:6826/1283302638,v1:172.21.15.156:6827/1283302638] exists,up b8c5bd1d-550a-410c-8247-2dc52f893b66 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.a [2,11] 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.10 [8,11] 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.14 [2,4] 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1 [4,9] 2024-04-25T01:27:09.488 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.3 [4,9] 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.19 [3,7] 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.21 [4,8] 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.25 [5,9] 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.27 [4,6] 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.28 [5,8] 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/1190188268 expires 2024-04-26T01:17:59.526723+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/4006719933 expires 2024-04-26T01:17:59.526723+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2073759739 expires 2024-04-26T01:17:59.526723+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/1190188268 expires 2024-04-26T01:17:59.526723+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1590801448 expires 2024-04-26T01:17:31.392949+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/1662737541 expires 2024-04-26T01:17:31.392949+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2428618530 expires 2024-04-26T01:17:31.392949+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/1662737541 expires 2024-04-26T01:17:31.392949+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2674684320 expires 2024-04-26T01:17:31.392949+0000 2024-04-25T01:27:09.489 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/153993658 expires 2024-04-26T01:17:59.526723+0000 2024-04-25T01:27:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe48c0a2c20 msgr2=0x7fe48c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe48c0a2c20 0x7fe48c0a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0101960 tx=0x7fe494009190 comp rx=0 tx=0).stop 2024-04-25T01:27:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 msgr2=0x7fe4a0198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 0x7fe4a0198990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe49c0098e0 tx=0x7fe49c009580 comp rx=0 tx=0).stop 2024-04-25T01:27:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 shutdown_connections 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe48c0a2c20 0x7fe48c0a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4a01024a0 0x7fe4a019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4a0100900 0x7fe4a0198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 --2- 172.21.15.16:0/793222498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4a0101b00 0x7fe4a0198990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 >> 172.21.15.16:0/793222498 conn(0x7fe4a00fbec0 msgr2=0x7fe4a0109760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 shutdown_connections 2024-04-25T01:27:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:09.509+0000 7fe4a85bb700 1 -- 172.21.15.16:0/793222498 wait complete. 2024-04-25T01:27:09.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:09 smithi016 ceph-mon[29153]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:09.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:09 smithi038 ceph-mon[32841]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:09.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:09 smithi156 ceph-mon[32631]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:10.371 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T01:27:10.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/1465761652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84103360 msgr2=0x7f1c84109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/1465761652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84103360 0x7f1c84109bf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c009ca0 tx=0x7f1c7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/1465761652 shutdown_connections 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/1465761652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c84103360 0x7f1c84109bf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/1465761652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c841029a0 0x7f1c84102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/1465761652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c841017b0 0x7f1c84101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/1465761652 >> 172.21.15.16:0/1465761652 conn(0x7f1c840fcd40 msgr2=0x7f1c840ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/1465761652 shutdown_connections 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.518+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/1465761652 wait complete. 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 Processor -- start 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 -- start start 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 0x7f1c840754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c841029a0 0x7f1c84073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c84103360 0x7f1c84074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8410d3a0 con 0x7f1c841017b0 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8410d220 con 0x7f1c84103360 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8b9b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8410d520 con 0x7f1c841029a0 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8974c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 0x7f1c840754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8974c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 0x7f1c840754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41784/0 (socket says 172.21.15.16:41784) 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8974c640 1 -- 172.21.15.16:0/2262949998 learned_addr learned my addr 172.21.15.16:0/2262949998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c88f4b640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c841029a0 0x7f1c84073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c89f4d640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c84103360 0x7f1c84074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8974c640 1 -- 172.21.15.16:0/2262949998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c841029a0 msgr2=0x7f1c84073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.519+0000 7f1c8974c640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c841029a0 0x7f1c84073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c8974c640 1 -- 172.21.15.16:0/2262949998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c84103360 msgr2=0x7f1c84074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c8974c640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c84103360 0x7f1c84074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c8974c640 1 -- 172.21.15.16:0/2262949998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c84074900 con 0x7f1c841017b0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c89f4d640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c84103360 0x7f1c84074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c8974c640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 0x7f1c840754f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000b450 tx=0x7f1c8000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80015020 con 0x7f1c841017b0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c80011d40 con 0x7f1c841017b0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c841a1ae0 con 0x7f1c841017b0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8000f340 con 0x7f1c841017b0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.520+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c841a2080 con 0x7f1c841017b0 2024-04-25T01:27:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.521+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c84108a70 con 0x7f1c841017b0 2024-04-25T01:27:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.522+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1c80018020 con 0x7f1c841017b0 2024-04-25T01:27:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.523+0000 7f1c7a7fc640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1c700a2c10 0x7f1c700a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.523+0000 7f1c88f4b640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1c700a2c10 0x7f1c700a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.523+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1c800cf0f0 con 0x7f1c841017b0 2024-04-25T01:27:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.523+0000 7f1c88f4b640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1c700a2c10 0x7f1c700a50d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1c84075260 tx=0x7f1c7400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.526+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1c8001f020 con 0x7f1c841017b0 2024-04-25T01:27:10.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:10 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/793222498' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:27:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.714+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c8402d3c0 con 0x7f1c841017b0 2024-04-25T01:27:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.715+0000 7f1c7a7fc640 1 -- 172.21.15.16:0/2262949998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7f1c8000d040 con 0x7f1c841017b0 2024-04-25T01:27:10.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:10.695 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.717+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1c700a2c10 msgr2=0x7f1c700a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.717+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1c700a2c10 0x7f1c700a50d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1c84075260 tx=0x7f1c7400a4c0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.717+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 msgr2=0x7f1c840754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.717+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 0x7f1c840754f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000b450 tx=0x7f1c8000b910 comp rx=0 tx=0).stop 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.717+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 shutdown_connections 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.717+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1c700a2c10 0x7f1c700a50d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.718+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c84103360 0x7f1c84074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.718+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c841029a0 0x7f1c84073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.718+0000 7f1c8b9b0640 1 --2- 172.21.15.16:0/2262949998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c841017b0 0x7f1c840754f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.718+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 >> 172.21.15.16:0/2262949998 conn(0x7f1c840fcd40 msgr2=0x7f1c84105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.718+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 shutdown_connections 2024-04-25T01:27:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.718+0000 7f1c8b9b0640 1 -- 172.21.15.16:0/2262949998 wait complete. 2024-04-25T01:27:10.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:27:10.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:27:10.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:10 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/793222498' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:27:10.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:10 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/793222498' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:27:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 -- 172.21.15.16:0/1794126685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac101890 msgr2=0x7f1fac101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/1794126685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac101890 0x7f1fac101d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009900 tx=0x7f1fa002a410 comp rx=0 tx=0).stop 2024-04-25T01:27:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 -- 172.21.15.16:0/1794126685 shutdown_connections 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/1794126685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 0x7f1fac108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/1794126685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac101890 0x7f1fac101d10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/1794126685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fac1006a0 0x7f1fac100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 -- 172.21.15.16:0/1794126685 >> 172.21.15.16:0/1794126685 conn(0x7f1fac0fbc30 msgr2=0x7f1fac0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 -- 172.21.15.16:0/1794126685 shutdown_connections 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.836+0000 7f1fb2a88640 1 -- 172.21.15.16:0/1794126685 wait complete. 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 Processor -- start 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 -- start start 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 0x7f1fac104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fac101890 0x7f1fac102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 0x7f1fac1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac10c1d0 con 0x7f1fac1006a0 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac10c050 con 0x7f1fac102250 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb2a88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac10c350 con 0x7f1fac101890 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb0824640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 0x7f1fac104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb0824640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 0x7f1fac104590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41804/0 (socket says 172.21.15.16:41804) 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb0824640 1 -- 172.21.15.16:0/3877351492 learned_addr learned my addr 172.21.15.16:0/3877351492 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.837+0000 7f1fb1025640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 0x7f1fac1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fabfff640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fac101890 0x7f1fac102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb0824640 1 -- 172.21.15.16:0/3877351492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fac101890 msgr2=0x7f1fac102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb0824640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fac101890 0x7f1fac102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb0824640 1 -- 172.21.15.16:0/3877351492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 msgr2=0x7f1fac1031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb0824640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 0x7f1fac1031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb0824640 1 -- 172.21.15.16:0/3877351492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac1039b0 con 0x7f1fac1006a0 2024-04-25T01:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb1025640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 0x7f1fac1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb0824640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 0x7f1fac104590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00c970 tx=0x7f1f9c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c004220 con 0x7f1fac1006a0 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f9c0043c0 con 0x7f1fac1006a0 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fac1a1b40 con 0x7f1fac1006a0 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c004a00 con 0x7f1fac1006a0 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.838+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fac1a2060 con 0x7f1fac1006a0 2024-04-25T01:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.839+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fac1a2290 con 0x7f1fac1006a0 2024-04-25T01:27:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.840+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1f9c0079b0 con 0x7f1fac1006a0 2024-04-25T01:27:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.841+0000 7f1fa9ffb640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1f940a2c10 0x7f1f940a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.841+0000 7f1fabfff640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1f940a2c10 0x7f1f940a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.841+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1f9c0cee70 con 0x7f1fac1006a0 2024-04-25T01:27:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.841+0000 7f1fabfff640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1f940a2c10 0x7f1f940a50d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1fa002a930 tx=0x7f1fa0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:10.845+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1f9c014030 con 0x7f1fac1006a0 2024-04-25T01:27:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.027+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1fac1a29a0 con 0x7f1fac1006a0 2024-04-25T01:27:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.027+0000 7f1fa9ffb640 1 -- 172.21.15.16:0/3877351492 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f1f9c09bd50 con 0x7f1fac1006a0 2024-04-25T01:27:11.007 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:27:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1f940a2c10 msgr2=0x7f1f940a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1f940a2c10 0x7f1f940a50d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1fa002a930 tx=0x7f1fa0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 msgr2=0x7f1fac104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 0x7f1fac104590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00c970 tx=0x7f1f9c00ce30 comp rx=0 tx=0).stop 2024-04-25T01:27:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 shutdown_connections 2024-04-25T01:27:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1f940a2c10 0x7f1f940a50d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fac102250 0x7f1fac1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fac101890 0x7f1fac102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 --2- 172.21.15.16:0/3877351492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fac1006a0 0x7f1fac104590 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.029+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 >> 172.21.15.16:0/3877351492 conn(0x7f1fac0fbc30 msgr2=0x7f1fac069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.030+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 shutdown_connections 2024-04-25T01:27:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.030+0000 7f1fb2a88640 1 -- 172.21.15.16:0/3877351492 wait complete. 2024-04-25T01:27:11.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:27:11.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 -- 172.21.15.16:0/4050694270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e041012c0 msgr2=0x7f1e04108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 --2- 172.21.15.16:0/4050694270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e041012c0 0x7f1e04108b60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009ca0 tx=0x7f1dfc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 -- 172.21.15.16:0/4050694270 shutdown_connections 2024-04-25T01:27:11.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 --2- 172.21.15.16:0/4050694270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e041012c0 0x7f1e04108b60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 --2- 172.21.15.16:0/4050694270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04100900 0x7f1e04100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 --2- 172.21.15.16:0/4050694270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04101cb0 0x7f1e041020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 -- 172.21.15.16:0/4050694270 >> 172.21.15.16:0/4050694270 conn(0x7f1e040fb370 msgr2=0x7f1e040fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.146+0000 7f1e09c45640 1 -- 172.21.15.16:0/4050694270 shutdown_connections 2024-04-25T01:27:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 -- 172.21.15.16:0/4050694270 wait complete. 2024-04-25T01:27:11.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 Processor -- start 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 -- start start 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 0x7f1e041978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e041012c0 0x7f1e04197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04101cb0 0x7f1e0419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410c1c0 con 0x7f1e04100900 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410c040 con 0x7f1e041012c0 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e09c45640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410c340 con 0x7f1e04101cb0 2024-04-25T01:27:11.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 0x7f1e041978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 0x7f1e041978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41828/0 (socket says 172.21.15.16:41828) 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.147+0000 7f1e037fe640 1 -- 172.21.15.16:0/1948282564 learned_addr learned my addr 172.21.15.16:0/1948282564 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e02ffd640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e041012c0 0x7f1e04197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e03fff640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04101cb0 0x7f1e0419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e037fe640 1 -- 172.21.15.16:0/1948282564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04101cb0 msgr2=0x7f1e0419ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e037fe640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04101cb0 0x7f1e0419ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e037fe640 1 -- 172.21.15.16:0/1948282564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e041012c0 msgr2=0x7f1e04197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e037fe640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e041012c0 0x7f1e04197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e037fe640 1 -- 172.21.15.16:0/1948282564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0419f4e0 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e037fe640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 0x7f1e041978c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1df400c970 tx=0x7f1df400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4007c10 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df4007db0 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0419f7d0 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df400f870 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.148+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e0419fd20 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.149+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e0419ffe0 con 0x7f1e04100900 2024-04-25T01:27:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.150+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1df400fa10 con 0x7f1e04100900 2024-04-25T01:27:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.151+0000 7f1e00ff9640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1dec0a2b90 0x7f1dec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.151+0000 7f1e02ffd640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1dec0a2b90 0x7f1dec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.151+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1df40cf460 con 0x7f1e04100900 2024-04-25T01:27:11.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.151+0000 7f1e02ffd640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1dec0a2b90 0x7f1dec0a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e04198dd0 tx=0x7f1df800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:11.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.155+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1df409c3d0 con 0x7f1e04100900 2024-04-25T01:27:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.339+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1e041a01b0 con 0x7f1e04100900 2024-04-25T01:27:11.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.339+0000 7f1e00ff9640 1 -- 172.21.15.16:0/1948282564 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f1df409bcd0 con 0x7f1e04100900 2024-04-25T01:27:11.319 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:27:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1dec0a2b90 msgr2=0x7f1dec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1dec0a2b90 0x7f1dec0a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e04198dd0 tx=0x7f1df800b390 comp rx=0 tx=0).stop 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 msgr2=0x7f1e041978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 0x7f1e041978c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1df400c970 tx=0x7f1df400ce30 comp rx=0 tx=0).stop 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 shutdown_connections 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1dec0a2b90 0x7f1dec0a5050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04101cb0 0x7f1e0419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e041012c0 0x7f1e04197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 --2- 172.21.15.16:0/1948282564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e04100900 0x7f1e041978c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 >> 172.21.15.16:0/1948282564 conn(0x7f1e040fb370 msgr2=0x7f1e040fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 shutdown_connections 2024-04-25T01:27:11.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.341+0000 7f1e09c45640 1 -- 172.21.15.16:0/1948282564 wait complete. 2024-04-25T01:27:11.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:27:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 -- 172.21.15.16:0/3027425498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a01006a0 msgr2=0x7f70a0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 --2- 172.21.15.16:0/3027425498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a01006a0 0x7f70a0100ac0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7090009900 tx=0x7f709002a410 comp rx=0 tx=0).stop 2024-04-25T01:27:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 -- 172.21.15.16:0/3027425498 shutdown_connections 2024-04-25T01:27:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 --2- 172.21.15.16:0/3027425498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a0102250 0x7f70a0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 --2- 172.21.15.16:0/3027425498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70a0101890 0x7f70a0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 --2- 172.21.15.16:0/3027425498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a01006a0 0x7f70a0100ac0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 -- 172.21.15.16:0/3027425498 >> 172.21.15.16:0/3027425498 conn(0x7f70a00fbc30 msgr2=0x7f70a00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 -- 172.21.15.16:0/3027425498 shutdown_connections 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.457+0000 7f70a6763640 1 -- 172.21.15.16:0/3027425498 wait complete. 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 Processor -- start 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 -- start start 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a01006a0 0x7f70a006a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 0x7f70a019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70a0102250 0x7f70a019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70a010c220 con 0x7f70a0101890 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f70a010c0a0 con 0x7f70a01006a0 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f70a6763640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70a010c3a0 con 0x7f70a0102250 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 0x7f70a019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 0x7f70a019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41844/0 (socket says 172.21.15.16:41844) 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 -- 172.21.15.16:0/1702753417 learned_addr learned my addr 172.21.15.16:0/1702753417 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709ffff640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a01006a0 0x7f70a006a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 -- 172.21.15.16:0/1702753417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70a0102250 msgr2=0x7f70a019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70a0102250 0x7f70a019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 -- 172.21.15.16:0/1702753417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a01006a0 msgr2=0x7f70a006a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a01006a0 0x7f70a006a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.458+0000 7f709f7fe640 1 -- 172.21.15.16:0/1702753417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a019f690 con 0x7f70a0101890 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.459+0000 7f709f7fe640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 0x7f70a019c800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f709400b4a0 tx=0x7f709400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.459+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094014070 con 0x7f70a0101890 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.459+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a019f980 con 0x7f70a0101890 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.459+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7094010ac0 con 0x7f70a0101890 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.459+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094017c30 con 0x7f70a0101890 2024-04-25T01:27:11.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.459+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a019ffa0 con 0x7f70a0101890 2024-04-25T01:27:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.460+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a0107b30 con 0x7f70a0101890 2024-04-25T01:27:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.461+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7094017070 con 0x7f70a0101890 2024-04-25T01:27:11.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.462+0000 7f709d7fa640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70880a2b90 0x7f70880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.462+0000 7f709ffff640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70880a2b90 0x7f70880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.462+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f70940ce740 con 0x7f70a0101890 2024-04-25T01:27:11.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.462+0000 7f709ffff640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70880a2b90 0x7f70880a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7090004770 tx=0x7f7090000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:11.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.466+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f709409b680 con 0x7f70a0101890 2024-04-25T01:27:11.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:11 smithi016 ceph-mon[29153]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:11.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:11 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2262949998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:11.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:11 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3877351492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:27:11.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:11 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1948282564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:27:11.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.652+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f70a0062340 con 0x7f70a0101890 2024-04-25T01:27:11.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.652+0000 7f709d7fa640 1 -- 172.21.15.16:0/1702753417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f709409af80 con 0x7f70a0101890 2024-04-25T01:27:11.631 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70880a2b90 msgr2=0x7f70880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70880a2b90 0x7f70880a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7090004770 tx=0x7f7090000bc0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 msgr2=0x7f70a019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 0x7f70a019c800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f709400b4a0 tx=0x7f709400b960 comp rx=0 tx=0).stop 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 shutdown_connections 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f70880a2b90 0x7f70880a5050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70a0102250 0x7f70a019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70a0101890 0x7f70a019c800 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 --2- 172.21.15.16:0/1702753417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70a01006a0 0x7f70a006a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 >> 172.21.15.16:0/1702753417 conn(0x7f70a00fbc30 msgr2=0x7f70a0104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 shutdown_connections 2024-04-25T01:27:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.654+0000 7f70a6763640 1 -- 172.21.15.16:0/1702753417 wait complete. 2024-04-25T01:27:11.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:27:11.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:11 smithi038 ceph-mon[32841]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:11.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:11 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2262949998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:11.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:11 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3877351492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:27:11.716 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:11 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1948282564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:27:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 -- 172.21.15.16:0/2580551549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc100680 msgr2=0x7fcadc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2580551549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc100680 0x7fcadc100aa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009900 tx=0x7fcacc02a410 comp rx=0 tx=0).stop 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 -- 172.21.15.16:0/2580551549 shutdown_connections 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2580551549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcadc102230 0x7fcadc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2580551549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcadc101870 0x7fcadc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2580551549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc100680 0x7fcadc100aa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 -- 172.21.15.16:0/2580551549 >> 172.21.15.16:0/2580551549 conn(0x7fcadc0fbbf0 msgr2=0x7fcadc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 -- 172.21.15.16:0/2580551549 shutdown_connections 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.767+0000 7fcae26cd640 1 -- 172.21.15.16:0/2580551549 wait complete. 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 Processor -- start 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 -- start start 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcadc100680 0x7fcadc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 0x7fcadc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcadc102230 0x7fcadc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc10c180 con 0x7fcadc101870 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc10c000 con 0x7fcadc102230 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae26cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc10c300 con 0x7fcadc100680 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcadb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 0x7fcadc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcadb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 0x7fcadc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47262/0 (socket says 172.21.15.16:47262) 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcadb7fe640 1 -- 172.21.15.16:0/2768014605 learned_addr learned my addr 172.21.15.16:0/2768014605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcae0c6a640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcadc102230 0x7fcadc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.768+0000 7fcadbfff640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcadc100680 0x7fcadc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcadb7fe640 1 -- 172.21.15.16:0/2768014605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcadc100680 msgr2=0x7fcadc197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcadb7fe640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcadc100680 0x7fcadc197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcadb7fe640 1 -- 172.21.15.16:0/2768014605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcadc102230 msgr2=0x7fcadc19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcadb7fe640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcadc102230 0x7fcadc19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcadb7fe640 1 -- 172.21.15.16:0/2768014605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcadc19f650 con 0x7fcadc101870 2024-04-25T01:27:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcadb7fe640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 0x7fcadc197f70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcac400d950 tx=0x7fcac400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4017030 con 0x7fcadc101870 2024-04-25T01:27:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcac4004020 con 0x7fcadc101870 2024-04-25T01:27:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcadc19f940 con 0x7fcadc101870 2024-04-25T01:27:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac40047a0 con 0x7fcadc101870 2024-04-25T01:27:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.769+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcadc19fe90 con 0x7fcadc101870 2024-04-25T01:27:11.751 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:11 smithi156 ceph-mon[32631]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:11 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2262949998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:11 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3877351492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:27:11.752 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:11 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1948282564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:27:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.770+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcadc1a0120 con 0x7fcadc101870 2024-04-25T01:27:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.771+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcac40041d0 con 0x7fcadc101870 2024-04-25T01:27:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.772+0000 7fcad97fa640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcac80a2b90 0x7fcac80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.772+0000 7fcadbfff640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcac80a2b90 0x7fcac80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.772+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcac40ceb40 con 0x7fcadc101870 2024-04-25T01:27:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.772+0000 7fcadbfff640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcac80a2b90 0x7fcac80a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcacc000c00 tx=0x7fcacc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:11.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:11.777+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcac409ba80 con 0x7fcadc101870 2024-04-25T01:27:11.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.001+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcadc1a02f0 con 0x7fcadc101870 2024-04-25T01:27:11.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.002+0000 7fcad97fa640 1 -- 172.21.15.16:0/2768014605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11866 (secure 0 0 0) 0x7fcac409b380 con 0x7fcadc101870 2024-04-25T01:27:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:11.981 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:25:21.068325+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24476,24499,24524]},"id":1}]} 2024-04-25T01:27:11.982 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:27:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcac80a2b90 msgr2=0x7fcac80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcac80a2b90 0x7fcac80a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcacc000c00 tx=0x7fcacc0314a0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 msgr2=0x7fcadc197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 0x7fcadc197f70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcac400d950 tx=0x7fcac400de10 comp rx=0 tx=0).stop 2024-04-25T01:27:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 shutdown_connections 2024-04-25T01:27:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcac80a2b90 0x7fcac80a5050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcadc102230 0x7fcadc19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcadc101870 0x7fcadc197f70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 --2- 172.21.15.16:0/2768014605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcadc100680 0x7fcadc197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 >> 172.21.15.16:0/2768014605 conn(0x7fcadc0fbbf0 msgr2=0x7fcadc068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 shutdown_connections 2024-04-25T01:27:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.004+0000 7fcae26cd640 1 -- 172.21.15.16:0/2768014605 wait complete. 2024-04-25T01:27:11.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/601885435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 msgr2=0x7fdfbc1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/601885435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc1045f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009900 tx=0x7fdfb402a410 comp rx=0 tx=0).stop 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/601885435 shutdown_connections 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/601885435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfbc107460 0x7fdfbc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/601885435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfbc104b30 0x7fdfbc106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/601885435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc1045f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/601885435 >> 172.21.15.16:0/601885435 conn(0x7fdfbc0fbbf0 msgr2=0x7fdfbc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/601885435 shutdown_connections 2024-04-25T01:27:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/601885435 wait complete. 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 Processor -- start 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 -- start start 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.118+0000 7fdfc3fa2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfbc104b30 0x7fdfbc1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfbc107460 0x7fdfbc1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc10d3f0 con 0x7fdfbc102200 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc10d270 con 0x7fdfbc104b30 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc10d570 con 0x7fdfbc107460 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc101b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47280/0 (socket says 172.21.15.16:47280) 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 -- 172.21.15.16:0/3181753673 learned_addr learned my addr 172.21.15.16:0/3181753673 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 -- 172.21.15.16:0/3181753673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfbc107460 msgr2=0x7fdfbc1006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfbc107460 0x7fdfbc1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 -- 172.21.15.16:0/3181753673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfbc104b30 msgr2=0x7fdfbc1001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfbc104b30 0x7fdfbc1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 -- 172.21.15.16:0/3181753673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfbc100fb0 con 0x7fdfbc102200 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc1d3e640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc101b20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40314e0 tx=0x7fdfb4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb403f070 con 0x7fdfbc102200 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfb4031e40 con 0x7fdfbc102200 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfbc1a1c50 con 0x7fdfbc102200 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4035030 con 0x7fdfbc102200 2024-04-25T01:27:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.119+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfbc1a21e0 con 0x7fdfbc102200 2024-04-25T01:27:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.120+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfbc0fd6e0 con 0x7fdfbc102200 2024-04-25T01:27:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.121+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdfb4042070 con 0x7fdfbc102200 2024-04-25T01:27:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.122+0000 7fdfbaffd640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfa40a2c10 0x7fdfa40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.122+0000 7fdfc153d640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfa40a2c10 0x7fdfa40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.122+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdfb40efe10 con 0x7fdfbc102200 2024-04-25T01:27:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.122+0000 7fdfc153d640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfa40a2c10 0x7fdfa40a50d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc101890 tx=0x7fdfb0006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.125+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfb40bcd50 con 0x7fdfbc102200 2024-04-25T01:27:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.312+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdfbc02d3c0 con 0x7fdfbc102200 2024-04-25T01:27:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.314+0000 7fdfbaffd640 1 -- 172.21.15.16:0/3181753673 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7fdfb40bc650 con 0x7fdfbc102200 2024-04-25T01:27:12.293 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:12.293 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:27:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfa40a2c10 msgr2=0x7fdfa40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfa40a2c10 0x7fdfa40a50d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc101890 tx=0x7fdfb0006c90 comp rx=0 tx=0).stop 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 msgr2=0x7fdfbc101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc101b20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40314e0 tx=0x7fdfb4003d50 comp rx=0 tx=0).stop 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 shutdown_connections 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdfa40a2c10 0x7fdfa40a50d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfbc107460 0x7fdfbc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfbc104b30 0x7fdfbc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 --2- 172.21.15.16:0/3181753673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfbc102200 0x7fdfbc101b20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 >> 172.21.15.16:0/3181753673 conn(0x7fdfbc0fbbf0 msgr2=0x7fdfbc102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 shutdown_connections 2024-04-25T01:27:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.316+0000 7fdfc3fa2640 1 -- 172.21.15.16:0/3181753673 wait complete. 2024-04-25T01:27:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:27:12.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:27:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 -- 172.21.15.16:0/1859164442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 msgr2=0x7f4074109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 --2- 172.21.15.16:0/1859164442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074109be0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f406c009ca0 tx=0x7f406c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 -- 172.21.15.16:0/1859164442 shutdown_connections 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 --2- 172.21.15.16:0/1859164442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074109be0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 --2- 172.21.15.16:0/1859164442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4074102990 0x7f4074102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 --2- 172.21.15.16:0/1859164442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40741017a0 0x7f4074101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 -- 172.21.15.16:0/1859164442 >> 172.21.15.16:0/1859164442 conn(0x7f40740fcd10 msgr2=0x7f40740ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 -- 172.21.15.16:0/1859164442 shutdown_connections 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 -- 172.21.15.16:0/1859164442 wait complete. 2024-04-25T01:27:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.433+0000 7f4079110640 1 Processor -- start 2024-04-25T01:27:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 -- start start 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40741017a0 0x7f40740754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4074102990 0x7f4074073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f407410d300 con 0x7f4074103350 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f407410d180 con 0x7f4074102990 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4079110640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f407410d480 con 0x7f40741017a0 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47298/0 (socket says 172.21.15.16:47298) 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 -- 172.21.15.16:0/1715490481 learned_addr learned my addr 172.21.15.16:0/1715490481 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f4072d9d640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40741017a0 0x7f40740754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407259c640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4074102990 0x7f4074073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 -- 172.21.15.16:0/1715490481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40741017a0 msgr2=0x7f40740754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40741017a0 0x7f40740754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 -- 172.21.15.16:0/1715490481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4074102990 msgr2=0x7f4074073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4074102990 0x7f4074073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 -- 172.21.15.16:0/1715490481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4074074900 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407259c640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4074102990 0x7f4074073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.434+0000 7f407359e640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074074070 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f406c031040 tx=0x7f406c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.435+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c032ec0 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.435+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f406c0044f0 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.435+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40741a1aa0 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.435+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c030d10 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.435+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40741a1ff0 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.436+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4074108ae0 con 0x7f4074103350 2024-04-25T01:27:12.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.437+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f406c004040 con 0x7f4074103350 2024-04-25T01:27:12.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.437+0000 7f405bfff640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f405c0a2d30 0x7f405c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.437+0000 7f4072d9d640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f405c0a2d30 0x7f405c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.437+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f406c0f06f0 con 0x7f4074103350 2024-04-25T01:27:12.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.438+0000 7f4072d9d640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f405c0a2d30 0x7f405c0a51f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4064005fd0 tx=0x7f406400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:12.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.442+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f406c0bd630 con 0x7f4074103350 2024-04-25T01:27:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.626+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4074062340 con 0x7f4074103350 2024-04-25T01:27:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.626+0000 7f405bfff640 1 -- 172.21.15.16:0/1715490481 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f406c0bcf30 con 0x7f4074103350 2024-04-25T01:27:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:27:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f405c0a2d30 msgr2=0x7f405c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f405c0a2d30 0x7f405c0a51f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4064005fd0 tx=0x7f406400b410 comp rx=0 tx=0).stop 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 msgr2=0x7f4074074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074074070 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f406c031040 tx=0x7f406c003ba0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 shutdown_connections 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f405c0a2d30 0x7f405c0a51f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4074103350 0x7f4074074070 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4074102990 0x7f4074073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 --2- 172.21.15.16:0/1715490481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40741017a0 0x7f40740754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 >> 172.21.15.16:0/1715490481 conn(0x7f40740fcd10 msgr2=0x7f4074105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 shutdown_connections 2024-04-25T01:27:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.628+0000 7f4079110640 1 -- 172.21.15.16:0/1715490481 wait complete. 2024-04-25T01:27:12.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:12 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1702753417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:27:12.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:12 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2768014605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:12.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:12 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3181753673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:12.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:27:12.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 -- 172.21.15.16:0/3227214477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8081033e0 msgr2=0x7fb808109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/3227214477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8081033e0 0x7fb808109c70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb800009cd0 tx=0x7fb80002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 -- 172.21.15.16:0/3227214477 shutdown_connections 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/3227214477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8081033e0 0x7fb808109c70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/3227214477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb808102a20 0x7fb808102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/3227214477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8080ffa50 0x7fb8080ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 -- 172.21.15.16:0/3227214477 >> 172.21.15.16:0/3227214477 conn(0x7fb8080fb380 msgr2=0x7fb8080fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 -- 172.21.15.16:0/3227214477 shutdown_connections 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.744+0000 7fb80e6a5640 1 -- 172.21.15.16:0/3227214477 wait complete. 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 Processor -- start 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 -- start start 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8080ffa50 0x7fb8081979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 0x7fb808197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8081033e0 0x7fb80819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb80810d2b0 con 0x7fb808102a20 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb80810d130 con 0x7fb8081033e0 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80e6a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb80810d430 con 0x7fb8080ffa50 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 0x7fb808197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 0x7fb808197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47316/0 (socket says 172.21.15.16:47316) 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 -- 172.21.15.16:0/979982610 learned_addr learned my addr 172.21.15.16:0/979982610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb807fff640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8080ffa50 0x7fb8081979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb80cc42640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8081033e0 0x7fb80819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 -- 172.21.15.16:0/979982610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8080ffa50 msgr2=0x7fb8081979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8080ffa50 0x7fb8081979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 -- 172.21.15.16:0/979982610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8081033e0 msgr2=0x7fb80819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8081033e0 0x7fb80819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb8077fe640 1 -- 172.21.15.16:0/979982610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80819f600 con 0x7fb808102a20 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.745+0000 7fb807fff640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8080ffa50 0x7fb8081979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:12.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.746+0000 7fb8077fe640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 0x7fb808197f20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00d950 tx=0x7fb7fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:12.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.746+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc017030 con 0x7fb808102a20 2024-04-25T01:27:12.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.746+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7fc004020 con 0x7fb808102a20 2024-04-25T01:27:12.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.746+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80819f8f0 con 0x7fb808102a20 2024-04-25T01:27:12.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.746+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0047a0 con 0x7fb808102a20 2024-04-25T01:27:12.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.746+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80819fe90 con 0x7fb808102a20 2024-04-25T01:27:12.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.747+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb808108b70 con 0x7fb808102a20 2024-04-25T01:27:12.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.749+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb7fc0041d0 con 0x7fb808102a20 2024-04-25T01:27:12.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.749+0000 7fb8057fa640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:12.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.749+0000 7fb807fff640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:12.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.749+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb7fc0cf1c0 con 0x7fb808102a20 2024-04-25T01:27:12.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.750+0000 7fb807fff640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb808069b60 tx=0x7fb7f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:12.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.753+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb7fc09c180 con 0x7fb808102a20 2024-04-25T01:27:12.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:12 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1702753417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:27:12.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:12 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2768014605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:12.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:12 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3181753673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.937+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb80802d3c0 con 0x7fb808102a20 2024-04-25T01:27:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.937+0000 7fb8057fa640 1 -- 172.21.15.16:0/979982610 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fb7fc09ba80 con 0x7fb808102a20 2024-04-25T01:27:12.917 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb7f00a2c10 msgr2=0x7fb7f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb808069b60 tx=0x7fb7f8005f90 comp rx=0 tx=0).stop 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 msgr2=0x7fb808197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 0x7fb808197f20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00d950 tx=0x7fb7fc00de10 comp rx=0 tx=0).stop 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 shutdown_connections 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8081033e0 0x7fb80819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb808102a20 0x7fb808197f20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 --2- 172.21.15.16:0/979982610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8080ffa50 0x7fb8081979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 >> 172.21.15.16:0/979982610 conn(0x7fb8080fb380 msgr2=0x7fb808105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 shutdown_connections 2024-04-25T01:27:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:12.939+0000 7fb80e6a5640 1 -- 172.21.15.16:0/979982610 wait complete. 2024-04-25T01:27:12.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:27:12.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:12 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1702753417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:27:12.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:12 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2768014605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:12.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:12 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3181753673' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 -- 172.21.15.16:0/3644491813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 msgr2=0x7f8450108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 --2- 172.21.15.16:0/3644491813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f8450108ae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8448009ca0 tx=0x7f844802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 -- 172.21.15.16:0/3644491813 shutdown_connections 2024-04-25T01:27:13.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 --2- 172.21.15.16:0/3644491813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f8450108ae0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 --2- 172.21.15.16:0/3644491813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8450101890 0x7f8450101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 --2- 172.21.15.16:0/3644491813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84501006a0 0x7f8450100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 -- 172.21.15.16:0/3644491813 >> 172.21.15.16:0/3644491813 conn(0x7f84500fbc10 msgr2=0x7f84500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 -- 172.21.15.16:0/3644491813 shutdown_connections 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 -- 172.21.15.16:0/3644491813 wait complete. 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.056+0000 7f84566ee640 1 Processor -- start 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 -- start start 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84501006a0 0x7f8450104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8450101890 0x7f8450102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f84501031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f845010c1d0 con 0x7f8450102250 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f845010c050 con 0x7f8450101890 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f84566ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f845010c350 con 0x7f84501006a0 2024-04-25T01:27:13.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f84501031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f84501031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47330/0 (socket says 172.21.15.16:47330) 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 -- 172.21.15.16:0/1238248743 learned_addr learned my addr 172.21.15.16:0/1238248743 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f844f7fe640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8450101890 0x7f8450102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f844ffff640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84501006a0 0x7f8450104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 -- 172.21.15.16:0/1238248743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84501006a0 msgr2=0x7f8450104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84501006a0 0x7f8450104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 -- 172.21.15.16:0/1238248743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8450101890 msgr2=0x7f8450102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8450101890 0x7f8450102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.057+0000 7f8454c8b640 1 -- 172.21.15.16:0/1238248743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8450103a40 con 0x7f8450102250 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.058+0000 7f8454c8b640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f84501031b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8448031970 tx=0x7f8448003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.058+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448003d90 con 0x7f8450102250 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.058+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84480044f0 con 0x7f8450102250 2024-04-25T01:27:13.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.058+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448030bd0 con 0x7f8450102250 2024-04-25T01:27:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.058+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84501a1a90 con 0x7f8450102250 2024-04-25T01:27:13.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.058+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84501a1fe0 con 0x7f8450102250 2024-04-25T01:27:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.059+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84501a21b0 con 0x7f8450102250 2024-04-25T01:27:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.060+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8448004040 con 0x7f8450102250 2024-04-25T01:27:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.060+0000 7f844d7fa640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f84380a2b90 0x7f84380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.061+0000 7f844ffff640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f84380a2b90 0x7f84380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.061+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f84480f0430 con 0x7f8450102250 2024-04-25T01:27:13.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.061+0000 7f844ffff640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f84380a2b90 0x7f84380a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f84501016f0 tx=0x7f8440005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:13.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.065+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84480bd370 con 0x7f8450102250 2024-04-25T01:27:13.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.249+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84501a2380 con 0x7f8450102250 2024-04-25T01:27:13.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.249+0000 7f844d7fa640 1 -- 172.21.15.16:0/1238248743 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f84480bcc70 con 0x7f8450102250 2024-04-25T01:27:13.229 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f84380a2b90 msgr2=0x7f84380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f84380a2b90 0x7f84380a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f84501016f0 tx=0x7f8440005f90 comp rx=0 tx=0).stop 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 msgr2=0x7f84501031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f84501031b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8448031970 tx=0x7f8448003ba0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 shutdown_connections 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f84380a2b90 0x7f84380a5050 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8450102250 0x7f84501031b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8450101890 0x7f8450102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 --2- 172.21.15.16:0/1238248743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84501006a0 0x7f8450104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 >> 172.21.15.16:0/1238248743 conn(0x7f84500fbc10 msgr2=0x7f84500fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:13.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 shutdown_connections 2024-04-25T01:27:13.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.251+0000 7f84566ee640 1 -- 172.21.15.16:0/1238248743 wait complete. 2024-04-25T01:27:13.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:27:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 -- 172.21.15.16:0/3499643047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 msgr2=0x7f5304109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 --2- 172.21.15.16:0/3499643047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f5304109bf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009ca0 tx=0x7f52fc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 -- 172.21.15.16:0/3499643047 shutdown_connections 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 --2- 172.21.15.16:0/3499643047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f5304109bf0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 --2- 172.21.15.16:0/3499643047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041029a0 0x7f5304102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 --2- 172.21.15.16:0/3499643047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53041017b0 0x7f5304101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 -- 172.21.15.16:0/3499643047 >> 172.21.15.16:0/3499643047 conn(0x7f53040fcd40 msgr2=0x7f53040ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 -- 172.21.15.16:0/3499643047 shutdown_connections 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.365+0000 7f530a914640 1 -- 172.21.15.16:0/3499643047 wait complete. 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 Processor -- start 2024-04-25T01:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 -- start start 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041017b0 0x7f5304197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53041029a0 0x7f5304197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f530419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f530410d2e0 con 0x7f5304103360 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f530410d160 con 0x7f53041017b0 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f530a914640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f530410d460 con 0x7f53041029a0 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f530419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f530419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47354/0 (socket says 172.21.15.16:47354) 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 -- 172.21.15.16:0/1636303042 learned_addr learned my addr 172.21.15.16:0/1636303042 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f53037fe640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53041029a0 0x7f5304197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5303fff640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041017b0 0x7f5304197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 -- 172.21.15.16:0/1636303042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53041029a0 msgr2=0x7f5304197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53041029a0 0x7f5304197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 -- 172.21.15.16:0/1636303042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041017b0 msgr2=0x7f5304197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041017b0 0x7f5304197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5308eb1640 1 -- 172.21.15.16:0/1636303042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530419f5b0 con 0x7f5304103360 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.366+0000 7f5303fff640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041017b0 0x7f5304197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.367+0000 7f5308eb1640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f530419ef40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009dc0 tx=0x7f52fc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.367+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc032ec0 con 0x7f5304103360 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.367+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52fc0044f0 con 0x7f5304103360 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.367+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530419f840 con 0x7f5304103360 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.367+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc030d40 con 0x7f5304103360 2024-04-25T01:27:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.367+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530419fd90 con 0x7f5304103360 2024-04-25T01:27:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.368+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5304108b40 con 0x7f5304103360 2024-04-25T01:27:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.369+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f52fc004040 con 0x7f5304103360 2024-04-25T01:27:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.370+0000 7f53017fa640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f52ec0a2d30 0x7f52ec0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.370+0000 7f5303fff640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f52ec0a2d30 0x7f52ec0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.370+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f52fc0f06d0 con 0x7f5304103360 2024-04-25T01:27:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.370+0000 7f5303fff640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f52ec0a2d30 0x7f52ec0a51f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f52f40058c0 tx=0x7f52f4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:13.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.374+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f52fc0bd5b0 con 0x7f5304103360 2024-04-25T01:27:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.600+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f530402d290 con 0x7f5304103360 2024-04-25T01:27:13.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.600+0000 7f53017fa640 1 -- 172.21.15.16:0/1636303042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11866 (secure 0 0 0) 0x7f52fc0bceb0 con 0x7f5304103360 2024-04-25T01:27:13.579 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:13.580 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:25:21.068325+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24476,24499,24524]},"id":1}]} 2024-04-25T01:27:13.580 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 21 2024-04-25T01:27:13.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f52ec0a2d30 msgr2=0x7f52ec0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f52ec0a2d30 0x7f52ec0a51f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f52f40058c0 tx=0x7f52f4005850 comp rx=0 tx=0).stop 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 msgr2=0x7f530419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f530419ef40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009dc0 tx=0x7f52fc003ba0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 shutdown_connections 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f52ec0a2d30 0x7f52ec0a51f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5304103360 0x7f530419ef40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53041029a0 0x7f5304197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 --2- 172.21.15.16:0/1636303042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53041017b0 0x7f5304197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 >> 172.21.15.16:0/1636303042 conn(0x7f53040fcd40 msgr2=0x7f5304105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 shutdown_connections 2024-04-25T01:27:13.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.602+0000 7f530a914640 1 -- 172.21.15.16:0/1636303042 wait complete. 2024-04-25T01:27:13.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:13 smithi016 ceph-mon[29153]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:27:13.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:13 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1715490481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:13 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/979982610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.592 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:13 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1238248743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:27:13.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 -- 172.21.15.16:0/2996402644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874101790 msgr2=0x7fa874101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2996402644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874101790 0x7fa874101bb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa864009900 tx=0x7fa86402a410 comp rx=0 tx=0).stop 2024-04-25T01:27:13.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 -- 172.21.15.16:0/2996402644 shutdown_connections 2024-04-25T01:27:13.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2996402644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa874103340 0x7fa874109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2996402644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874102980 0x7fa874102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2996402644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874101790 0x7fa874101bb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 -- 172.21.15.16:0/2996402644 >> 172.21.15.16:0/2996402644 conn(0x7fa8740fcd20 msgr2=0x7fa8740ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 -- 172.21.15.16:0/2996402644 shutdown_connections 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.716+0000 7fa8794a5640 1 -- 172.21.15.16:0/2996402644 wait complete. 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 Processor -- start 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 -- start start 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa874101790 0x7fa8740754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 0x7fa874073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874103340 0x7fa874074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa87410d2b0 con 0x7fa874102980 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa87410d130 con 0x7fa874101790 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8794a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa87410d430 con 0x7fa874103340 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 0x7fa874073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 0x7fa874073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47364/0 (socket says 172.21.15.16:47364) 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 -- 172.21.15.16:0/2626905429 learned_addr learned my addr 172.21.15.16:0/2626905429 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa872ffd640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa874101790 0x7fa8740754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8737fe640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874103340 0x7fa874074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 -- 172.21.15.16:0/2626905429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874103340 msgr2=0x7fa874074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874103340 0x7fa874074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 -- 172.21.15.16:0/2626905429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa874101790 msgr2=0x7fa8740754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa874101790 0x7fa8740754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa86bfff640 1 -- 172.21.15.16:0/2626905429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa874074900 con 0x7fa874102980 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.717+0000 7fa8737fe640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874103340 0x7fa874074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.718+0000 7fa86bfff640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 0x7fa874073b30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00d770 tx=0x7fa85c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.718+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c017070 con 0x7fa874102980 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.718+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa85c0054d0 con 0x7fa874102980 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.718+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8741a1b00 con 0x7fa874102980 2024-04-25T01:27:13.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.718+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c004cf0 con 0x7fa874102980 2024-04-25T01:27:13.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.718+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8741a2050 con 0x7fa874102980 2024-04-25T01:27:13.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.719+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa874108b30 con 0x7fa874102980 2024-04-25T01:27:13.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.720+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa85c004e90 con 0x7fa874102980 2024-04-25T01:27:13.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.720+0000 7fa870ff9640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8600a2c10 0x7fa8600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:13.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.721+0000 7fa872ffd640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8600a2c10 0x7fa8600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:13.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.721+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa85c0cea50 con 0x7fa874102980 2024-04-25T01:27:13.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.721+0000 7fa872ffd640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8600a2c10 0x7fa8600a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa864004770 tx=0x7fa864000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:13.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.725+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa85c09b990 con 0x7fa874102980 2024-04-25T01:27:13.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:13 smithi156 ceph-mon[32631]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:27:13.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:13 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1715490481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:13 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/979982610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:13 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1238248743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.867 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:13 smithi038 ceph-mon[32841]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:27:13.867 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:13 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1715490481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.867 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:13 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/979982610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.867 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:13 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1238248743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:27:13.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.910+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa874062340 con 0x7fa874102980 2024-04-25T01:27:13.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.912+0000 7fa870ff9640 1 -- 172.21.15.16:0/2626905429 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7fa85c09b290 con 0x7fa874102980 2024-04-25T01:27:13.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:27:13.891 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:27:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8600a2c10 msgr2=0x7fa8600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8600a2c10 0x7fa8600a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa864004770 tx=0x7fa864000bc0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 msgr2=0x7fa874073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 0x7fa874073b30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00d770 tx=0x7fa85c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 shutdown_connections 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa8600a2c10 0x7fa8600a50d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa874103340 0x7fa874074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa874102980 0x7fa874073b30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 --2- 172.21.15.16:0/2626905429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa874101790 0x7fa8740754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 >> 172.21.15.16:0/2626905429 conn(0x7fa8740fcd20 msgr2=0x7fa874105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 shutdown_connections 2024-04-25T01:27:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:13.914+0000 7fa8794a5640 1 -- 172.21.15.16:0/2626905429 wait complete. 2024-04-25T01:27:13.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:25:16.327985+0000', 'modified': '2024-04-25T01:25:21.068325+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24499, 'mds_2': 24476}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6835/2628472521', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6834', 'nonce': 2628472521}, {'type': 'v1', 'addr': '172.21.15.38:6835', 'nonce': 2628472521}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/2186765223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 2186765223}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 2186765223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/2504948433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 2504948433}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 2504948433}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'h', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.38:6837/401186797', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6836', 'nonce': 401186797}, {'type': 'v1', 'addr': '172.21.15.38:6837', 'nonce': 401186797}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.156:6839/1428073868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1428073868}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1428073868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/2790744598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 2790744598}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 2790744598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24476, 24499, 24524]} 2024-04-25T01:27:13.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T01:27:13.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 -- 172.21.15.16:0/99111790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 msgr2=0x7f8050102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 --2- 172.21.15.16:0/99111790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050102e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8038009900 tx=0x7f803802a410 comp rx=0 tx=0).stop 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 -- 172.21.15.16:0/99111790 shutdown_connections 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 --2- 172.21.15.16:0/99111790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8050103350 0x7f8050109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 --2- 172.21.15.16:0/99111790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050102e10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 --2- 172.21.15.16:0/99111790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80501017a0 0x7f8050101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 -- 172.21.15.16:0/99111790 >> 172.21.15.16:0/99111790 conn(0x7f80500fcd10 msgr2=0x7f80500ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 -- 172.21.15.16:0/99111790 shutdown_connections 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.031+0000 7f80550fc640 1 -- 172.21.15.16:0/99111790 wait complete. 2024-04-25T01:27:14.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 Processor -- start 2024-04-25T01:27:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 -- start start 2024-04-25T01:27:14.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80501017a0 0x7f80501979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8050103350 0x7f805019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f805010d2b0 con 0x7f8050102990 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f805010d130 con 0x7f80501017a0 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f80550fc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f805010d430 con 0x7f8050103350 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.032+0000 7f804e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47384/0 (socket says 172.21.15.16:47384) 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 -- 172.21.15.16:0/1084620616 learned_addr learned my addr 172.21.15.16:0/1084620616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804ed9d640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80501017a0 0x7f80501979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804f59e640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8050103350 0x7f805019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 -- 172.21.15.16:0/1084620616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8050103350 msgr2=0x7f805019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8050103350 0x7f805019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 -- 172.21.15.16:0/1084620616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80501017a0 msgr2=0x7f80501979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80501017a0 0x7f80501979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 -- 172.21.15.16:0/1084620616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805019f5d0 con 0x7f8050102990 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f804e59c640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050197ef0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8038004770 tx=0x7f80380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f8047fff640 1 -- 172.21.15.16:0/1084620616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803803f030 con 0x7f8050102990 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f8047fff640 1 -- 172.21.15.16:0/1084620616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8038003cb0 con 0x7f8050102990 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805019f860 con 0x7f8050102990 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f8047fff640 1 -- 172.21.15.16:0/1084620616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038035070 con 0x7f8050102990 2024-04-25T01:27:14.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.033+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805019fd50 con 0x7f8050102990 2024-04-25T01:27:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8047fff640 1 -- 172.21.15.16:0/1084620616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8038042030 con 0x7f8050102990 2024-04-25T01:27:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 Processor -- start 2024-04-25T01:27:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 -- start start 2024-04-25T01:27:14.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f3579b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f3579be30 0x564f357a0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f357a0fa0 0x564f357a3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f356fef00 con 0x564f3579b440 2024-04-25T01:27:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564f356a7750 con 0x564f357a0fa0 2024-04-25T01:27:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f8057e07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564f356faec0 con 0x564f3579be30 2024-04-25T01:27:14.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f804f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f3579b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f804f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f3579b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47398/0 (socket says 172.21.15.16:47398) 2024-04-25T01:27:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.036+0000 7f804f59e640 1 -- 172.21.15.16:0/245093828 learned_addr learned my addr 172.21.15.16:0/245093828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804e59c640 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f357a0fa0 0x564f357a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f8047fff640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f803c0a2b90 0x7f803c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804ed9d640 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f3579be30 0x564f357a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 -- 172.21.15.16:0/245093828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f3579be30 msgr2=0x564f357a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f3579be30 0x564f357a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 -- 172.21.15.16:0/245093828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f357a0fa0 msgr2=0x564f357a3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f357a0fa0 0x564f357a3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 -- 172.21.15.16:0/245093828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f354132d0 con 0x564f3579b440 2024-04-25T01:27:14.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f8047fff640 1 -- 172.21.15.16:0/1084620616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f80380efa60 con 0x7f8050102990 2024-04-25T01:27:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f803c0a2b90 0x7f803c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f3579b860 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8048009fc0 tx=0x7f8048010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22658e416927085e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f80457fa640 1 -- 172.21.15.16:0/245093828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804801c900 con 0x564f3579b440 2024-04-25T01:27:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f80457fa640 1 -- 172.21.15.16:0/245093828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80480113e0 con 0x564f3579b440 2024-04-25T01:27:14.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f804f59e640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f803c0a2b90 0x7f803c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8048020000 tx=0x7f804801d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.037+0000 7f80457fa640 1 -- 172.21.15.16:0/245093828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048013620 con 0x564f3579b440 2024-04-25T01:27:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 -- 172.21.15.16:0/245093828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 msgr2=0x564f3579b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f3579b860 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8048009fc0 tx=0x7f8048010800 comp rx=0 tx=0).stop 2024-04-25T01:27:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 -- 172.21.15.16:0/245093828 shutdown_connections 2024-04-25T01:27:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f357a0fa0 0x564f357a3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f3579be30 0x564f357a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 --2- 172.21.15.16:0/245093828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f3579b860 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 -- 172.21.15.16:0/245093828 >> 172.21.15.16:0/245093828 conn(0x564f35796ab0 msgr2=0x564f35796eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 -- 172.21.15.16:0/245093828 shutdown_connections 2024-04-25T01:27:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.038+0000 7f8057e07740 1 -- 172.21.15.16:0/245093828 wait complete. 2024-04-25T01:27:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 -- start start 2024-04-25T01:27:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 Processor -- start 2024-04-25T01:27:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f357a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f357a3f50 con 0x564f3579be30 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564f356bacf0 con 0x564f3579b440 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564f357a40d0 con 0x564f357a0fa0 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f8057e07740 10 client.0 _ll_register_callbacks cb 0x564f356ba6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f357a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f357a5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47412/0 (socket says 172.21.15.16:47412) 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 -- 172.21.15.16:0/730836201 learned_addr learned my addr 172.21.15.16:0/730836201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804f59e640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 -- 172.21.15.16:0/730836201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 msgr2=0x564f357acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 -- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 msgr2=0x564f357a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804ed9d640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 -- 172.21.15.16:0/730836201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f354dd6b0 con 0x564f3579be30 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f804e59c640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f357a5e60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8038003fd0 tx=0x7f8038009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3214ec7332e4a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.039+0000 7f80367fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f80367fc640 1 -- 172.21.15.16:0/730836201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038049040 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f80367fc640 1 -- 172.21.15.16:0/730836201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f803803bc90 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80480113e0 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f80367fc640 1 -- 172.21.15.16:0/730836201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f80380d5050 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 10 client.15210 fetch_fsmap learned FSMap version 21 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564f357afe80 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f80367fc640 1 -- 172.21.15.16:0/730836201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80480113e0 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f80367fc640 1 -- 172.21.15.16:0/730836201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f803803b460 con 0x564f3579be30 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 10 client.15210 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 10 client.15210 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 20 client.15210 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f357b25c0 0x564f357b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 4 client.15210 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.040+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 --> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f3570c440 con 0x564f357b25c0 2024-04-25T01:27:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.041+0000 7f804ed9d640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f357b25c0 0x564f357b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.041+0000 7f804ed9d640 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f357b25c0 0x564f357b3a30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b02817416f7fff51 server_cookie=f93e5edaf6954e1a in_seq=0 out_seq=0 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.041+0000 7f80367fc640 10 client.15210 ms_handle_connect on v2:172.21.15.38:6836/401186797 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f80367fc640 1 -- 172.21.15.16:0/730836201 <== mds.2 v2:172.21.15.38:6836/401186797 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f3570c440 con 0x564f357b25c0 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f80367fc640 10 client.15210 handle_command_reply: tid=0 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 client.15210 shutdown 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 msgr2=0x564f357a5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f357a5e60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8038003fd0 tx=0x7f8038009620 comp rx=0 tx=0).stop 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 shutdown_connections 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f357b25c0 0x564f357b3a30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f357a5e60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 --2- 172.21.15.16:0/730836201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 >> 172.21.15.16:0/730836201 conn(0x564f35796ab0 msgr2=0x564f35795ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f80367fc640 0 client.15210 ms_handle_reset on v2:172.21.15.38:6836/401186797 2024-04-25T01:27:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 shutdown_connections 2024-04-25T01:27:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 1 -- 172.21.15.16:0/730836201 wait complete. 2024-04-25T01:27:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.042+0000 7f8057e07740 20 client.15210 trim_cache size 0 max 16384 2024-04-25T01:27:14.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.060+0000 7f8057e07740 1 Processor -- start 2024-04-25T01:27:14.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 -- start start 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f35791690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f357b1d70 con 0x564f3579be30 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564f357b2440 con 0x564f3579b440 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f8057e07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564f357b2b10 con 0x564f357a0fa0 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f804e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f35791690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f804e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f35791690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47420/0 (socket says 172.21.15.16:47420) 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f804e59c640 1 -- 172.21.15.16:0/2315425818 learned_addr learned my addr 172.21.15.16:0/2315425818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f804ed9d640 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.061+0000 7f804f59e640 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804e59c640 1 -- 172.21.15.16:0/2315425818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 msgr2=0x564f357b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804e59c640 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804e59c640 1 -- 172.21.15.16:0/2315425818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 msgr2=0x564f357a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804e59c640 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804e59c640 1 -- 172.21.15.16:0/2315425818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f357a0530 con 0x564f3579be30 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804ed9d640 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f804e59c640 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f35791690 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f80380b8c50 tx=0x7f80380ba790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96acb9bdf130e30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f8044ff9640 1 -- 172.21.15.16:0/2315425818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80380d9070 con 0x564f3579be30 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f8044ff9640 1 -- 172.21.15.16:0/2315425818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80380f3010 con 0x564f3579be30 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.062+0000 7f8044ff9640 1 -- 172.21.15.16:0/2315425818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803803b460 con 0x564f3579be30 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- 172.21.15.16:0/2315425818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 msgr2=0x564f35791690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f35791690 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f80380b8c50 tx=0x7f80380ba790 comp rx=0 tx=0).stop 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- 172.21.15.16:0/2315425818 shutdown_connections 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f357b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579be30 0x564f35791690 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- 172.21.15.16:0/2315425818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579b440 0x564f357a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- 172.21.15.16:0/2315425818 >> 172.21.15.16:0/2315425818 conn(0x564f35796ab0 msgr2=0x564f35796e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- 172.21.15.16:0/2315425818 shutdown_connections 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- 172.21.15.16:0/2315425818 wait complete. 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- start start 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 Processor -- start 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f357a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579be30 0x564f357a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f3581aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f357b38b0 con 0x564f3579b440 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564f357b31e0 con 0x564f3579be30 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564f357b3f80 con 0x564f357a0fa0 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f8057e07740 10 client.0 _ll_register_callbacks cb 0x7f8038003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.063+0000 7f804ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f357a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f357a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47428/0 (socket says 172.21.15.16:47428) 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 -- 172.21.15.16:0/1969091799 learned_addr learned my addr 172.21.15.16:0/1969091799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804e59c640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579be30 0x564f357a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804f59e640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f3581aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 -- 172.21.15.16:0/1969091799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 msgr2=0x564f3581aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f3581aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 -- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579be30 msgr2=0x564f357a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579be30 0x564f357a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 -- 172.21.15.16:0/1969091799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f3581b0a0 con 0x564f3579b440 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f804ed9d640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f357a7e80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8040000c00 tx=0x7f804000bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f569157bb68285d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f80367fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f80367fc640 1 -- 172.21.15.16:0/1969091799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804001bdc0 con 0x564f3579b440 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f35706790 con 0x564f3579b440 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f80367fc640 1 -- 172.21.15.16:0/1969091799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f80400035f0 con 0x564f3579b440 2024-04-25T01:27:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f80367fc640 1 -- 172.21.15.16:0/1969091799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f80400037f0 con 0x564f3579b440 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f8057e07740 10 client.15216 fetch_fsmap learned FSMap version 21 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564f3581c470 con 0x564f3579b440 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.064+0000 7f80367fc640 1 -- 172.21.15.16:0/1969091799 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564f35706790 con 0x564f3579b440 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f80367fc640 1 -- 172.21.15.16:0/1969091799 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f80400248f0 con 0x564f3579b440 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f8057e07740 10 client.15216 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f8057e07740 10 client.15216 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f8057e07740 20 client.15216 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f8057e07740 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f35821040 0x564f3581fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f8057e07740 4 client.15216 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 --> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] -- command(tid 0: {"prefix": "status"}) -- 0x564f35684b80 con 0x564f35821040 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.065+0000 7f804e59c640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f35821040 0x564f3581fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f80367fc640 10 client.15216 ms_handle_connect on v2:172.21.15.38:6836/401186797 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f804e59c640 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f35821040 0x564f3581fd50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ab6559906826647 server_cookie=4e5cd371a7098a49 in_seq=0 out_seq=0 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f80367fc640 1 -- 172.21.15.16:0/1969091799 <== mds.2 v2:172.21.15.38:6836/401186797 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564f35684b80 con 0x564f35821040 2024-04-25T01:27:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f80367fc640 10 client.15216 handle_command_reply: tid=0 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 client.15216 shutdown 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 msgr2=0x564f357a7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f357a7e80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8040000c00 tx=0x7f804000bcc0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 shutdown_connections 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:6836/401186797,v1:172.21.15.38:6837/401186797] conn(0x564f35821040 0x564f3581fd50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f357a0fa0 0x564f3581aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f3579be30 0x564f357a83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 --2- 172.21.15.16:0/1969091799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f3579b440 0x564f357a7e80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f80367fc640 0 client.15216 ms_handle_reset on v2:172.21.15.38:6836/401186797 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 >> 172.21.15.16:0/1969091799 conn(0x564f35796ab0 msgr2=0x564f35791690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 shutdown_connections 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 1 -- 172.21.15.16:0/1969091799 wait complete. 2024-04-25T01:27:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.066+0000 7f8057e07740 20 client.15216 trim_cache size 0 max 16384 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24476, 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 114.003272021, 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 21, 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 87, 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-04-25T01:27:14.054 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 135.02772902300001 2024-04-25T01:27:14.055 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:27:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f803c0a2b90 msgr2=0x7f803c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f803c0a2b90 0x7f803c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8048020000 tx=0x7f804801d000 comp rx=0 tx=0).stop 2024-04-25T01:27:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 msgr2=0x7f8050197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050197ef0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8038004770 tx=0x7f80380042f0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 shutdown_connections 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f803c0a2b90 0x7f803c0a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8050103350 0x7f805019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8050102990 0x7f8050197ef0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 --2- 172.21.15.16:0/1084620616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80501017a0 0x7f80501979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 >> 172.21.15.16:0/1084620616 conn(0x7f80500fcd10 msgr2=0x7f8050105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 shutdown_connections 2024-04-25T01:27:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.067+0000 7f80550fc640 1 -- 172.21.15.16:0/1084620616 wait complete. 2024-04-25T01:27:14.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:27:14.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 -- 172.21.15.16:0/880313708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290103340 msgr2=0x7f0290109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 --2- 172.21.15.16:0/880313708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290103340 0x7f0290109bd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0288009ca0 tx=0x7f028802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 -- 172.21.15.16:0/880313708 shutdown_connections 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 --2- 172.21.15.16:0/880313708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290103340 0x7f0290109bd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 --2- 172.21.15.16:0/880313708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290102980 0x7f0290102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 --2- 172.21.15.16:0/880313708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 0x7f0290101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 -- 172.21.15.16:0/880313708 >> 172.21.15.16:0/880313708 conn(0x7f02900fcd20 msgr2=0x7f02900ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 -- 172.21.15.16:0/880313708 shutdown_connections 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 -- 172.21.15.16:0/880313708 wait complete. 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 Processor -- start 2024-04-25T01:27:14.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.181+0000 7f0295396640 1 -- start start 2024-04-25T01:27:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0295396640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 0x7f02900754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0295396640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 0x7f0290073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0295396640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290103340 0x7f0290074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0295396640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f029010d2a0 con 0x7f0290102980 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0295396640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f029010d120 con 0x7f0290101790 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0295396640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f029010d420 con 0x7f0290103340 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 0x7f0290073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 0x7f0290073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47448/0 (socket says 172.21.15.16:47448) 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 -- 172.21.15.16:0/1453432276 learned_addr learned my addr 172.21.15.16:0/1453432276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028effd640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 0x7f02900754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028f7fe640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290103340 0x7f0290074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 -- 172.21.15.16:0/1453432276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290103340 msgr2=0x7f0290074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290103340 0x7f0290074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 -- 172.21.15.16:0/1453432276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 msgr2=0x7f02900754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 0x7f02900754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 -- 172.21.15.16:0/1453432276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0290074900 con 0x7f0290102980 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028effd640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 0x7f02900754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028f7fe640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290103340 0x7f0290074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 0x7f0290073b30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f028400d950 tx=0x7f028400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.182+0000 7f0277fff640 1 -- 172.21.15.16:0/1453432276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284017070 con 0x7f0290102980 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.183+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02901a1a80 con 0x7f0290102980 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.183+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02901a2020 con 0x7f0290102980 2024-04-25T01:27:14.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.183+0000 7f0277fff640 1 -- 172.21.15.16:0/1453432276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0284009c30 con 0x7f0290102980 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.183+0000 7f0277fff640 1 -- 172.21.15.16:0/1453432276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284020060 con 0x7f0290102980 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0277fff640 1 -- 172.21.15.16:0/1453432276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0284010460 con 0x7f0290102980 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 Processor -- start 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 -- start start 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8e30 0x5556858dd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 0x5556858e0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55568583bf00 con 0x5556858ddfa0 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5556857e4750 con 0x5556858d8440 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0298176740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555685837ec0 con 0x5556858d8e30 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 0x5556858e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 0x5556858e0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47458/0 (socket says 172.21.15.16:47458) 2024-04-25T01:27:14.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f028e7fc640 1 -- 172.21.15.16:0/1476125690 learned_addr learned my addr 172.21.15.16:0/1476125690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.185+0000 7f0277fff640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02780a2b90 0x7f02780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028f7fe640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02780a2b90 0x7f02780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028effd640 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8e30 0x5556858dd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028f7fe640 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f0277fff640 1 -- 172.21.15.16:0/1453432276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02840ce960 con 0x7f0290102980 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028e7fc640 1 -- 172.21.15.16:0/1476125690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8e30 msgr2=0x5556858dd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8e30 0x5556858dd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028e7fc640 1 -- 172.21.15.16:0/1476125690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 msgr2=0x5556858d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858d8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028e7fc640 1 -- 172.21.15.16:0/1476125690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556855502d0 con 0x5556858ddfa0 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028effd640 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8e30 0x5556858dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:14.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 0x5556858e0430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f028409b130 tx=0x7f028409b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7598265c599e2d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f028f7fe640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02780a2b90 0x7f02780a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x5556858d9690 tx=0x7f0288005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f02757fa640 1 -- 172.21.15.16:0/1476125690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028409b8a0 con 0x5556858ddfa0 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f02757fa640 1 -- 172.21.15.16:0/1476125690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0284009de0 con 0x5556858ddfa0 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.186+0000 7f02757fa640 1 -- 172.21.15.16:0/1476125690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284097810 con 0x5556858ddfa0 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- 172.21.15.16:0/1476125690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 msgr2=0x5556858e0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 0x5556858e0430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f028409b130 tx=0x7f028409b090 comp rx=0 tx=0).stop 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- 172.21.15.16:0/1476125690 shutdown_connections 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858ddfa0 0x5556858e0430 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8e30 0x5556858dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- 172.21.15.16:0/1476125690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- 172.21.15.16:0/1476125690 >> 172.21.15.16:0/1476125690 conn(0x5556858d3ab0 msgr2=0x5556858d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- 172.21.15.16:0/1476125690 shutdown_connections 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- 172.21.15.16:0/1476125690 wait complete. 2024-04-25T01:27:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- start start 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 Processor -- start 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858e27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556858e0ef0 con 0x5556858d8e30 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5556857f7cf0 con 0x5556858ddfa0 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5556858e1070 con 0x5556858d8440 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.187+0000 7f0298176740 10 client.0 _ll_register_callbacks cb 0x5556857f76e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47470/0 (socket says 172.21.15.16:47470) 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 -- 172.21.15.16:0/2799413487 learned_addr learned my addr 172.21.15.16:0/2799413487 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028effd640 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858e27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 -- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 msgr2=0x5556858e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858e27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 -- 172.21.15.16:0/2799413487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 msgr2=0x5556858e9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 -- 172.21.15.16:0/2799413487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55568561a6b0 con 0x5556858d8e30 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f028e7fc640 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e2d10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f02840058c0 tx=0x7f02840a3e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931f0e686b5c0629 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0272ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0272ff5640 1 -- 172.21.15.16:0/2799413487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028409f550 con 0x5556858d8e30 2024-04-25T01:27:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0284009de0 con 0x5556858d8e30 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0272ff5640 1 -- 172.21.15.16:0/2799413487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f028409f170 con 0x5556858d8e30 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0272ff5640 1 -- 172.21.15.16:0/2799413487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f028409a5f0 con 0x5556858d8e30 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0298176740 10 client.15228 fetch_fsmap learned FSMap version 21 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5556858ecd30 con 0x5556858d8e30 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.188+0000 7f0272ff5640 1 -- 172.21.15.16:0/2799413487 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0284009de0 con 0x5556858d8e30 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0272ff5640 1 -- 172.21.15.16:0/2799413487 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0284023060 con 0x5556858d8e30 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0298176740 10 client.15228 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0298176740 10 client.15228 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0298176740 20 client.15228 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0298176740 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x5556858eef70 0x5556858f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0298176740 4 client.15228 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 --> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555685849440 con 0x5556858eef70 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f028effd640 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x5556858eef70 0x5556858f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f0272ff5640 10 client.15228 ms_handle_connect on v2:172.21.15.156:6838/1428073868 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.189+0000 7f028effd640 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x5556858eef70 0x5556858f03e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=930e3b65ca84bfdf server_cookie=45dd16915476072c in_seq=0 out_seq=0 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0272ff5640 1 -- 172.21.15.16:0/2799413487 <== mds.1 v2:172.21.15.156:6838/1428073868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555685849440 con 0x5556858eef70 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0272ff5640 10 client.15228 handle_command_reply: tid=0 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 client.15228 shutdown 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 msgr2=0x5556858e2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e2d10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f02840058c0 tx=0x7f02840a3e10 comp rx=0 tx=0).stop 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 shutdown_connections 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x5556858eef70 0x5556858f03e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e2d10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 --2- 172.21.15.16:0/2799413487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858e27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0272ff5640 0 client.15228 ms_handle_reset on v2:172.21.15.156:6838/1428073868 2024-04-25T01:27:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 >> 172.21.15.16:0/2799413487 conn(0x5556858d3ab0 msgr2=0x5556858d2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 shutdown_connections 2024-04-25T01:27:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.190+0000 7f0298176740 1 -- 172.21.15.16:0/2799413487 wait complete. 2024-04-25T01:27:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.191+0000 7f0298176740 20 client.15228 trim_cache size 0 max 16384 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.208+0000 7f0298176740 1 Processor -- start 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.208+0000 7f0298176740 1 -- start start 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0298176740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858da190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0298176740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858da6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0298176740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0298176740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556858ee720 con 0x5556858d8e30 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0298176740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5556858eedf0 con 0x5556858ddfa0 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0298176740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5556858ef4c0 con 0x5556858d8440 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858da6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858da6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47476/0 (socket says 172.21.15.16:47476) 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 -- 172.21.15.16:0/1004327912 learned_addr learned my addr 172.21.15.16:0/1004327912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 -- 172.21.15.16:0/1004327912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 msgr2=0x5556858da190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858da190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 -- 172.21.15.16:0/1004327912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 msgr2=0x5556858e9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 -- 172.21.15.16:0/1004327912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556858d96c0 con 0x5556858d8e30 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f028e7fc640 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858da6d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0284097b70 tx=0x7f02840d20b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e949e86c415034a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0274ff9640 1 -- 172.21.15.16:0/1004327912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028409a410 con 0x5556858d8e30 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0274ff9640 1 -- 172.21.15.16:0/1004327912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02840b4070 con 0x5556858d8e30 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.209+0000 7f0274ff9640 1 -- 172.21.15.16:0/1004327912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028409f550 con 0x5556858d8e30 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- 172.21.15.16:0/1004327912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 msgr2=0x5556858da6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858da6d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0284097b70 tx=0x7f02840d20b0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- 172.21.15.16:0/1004327912 shutdown_connections 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858ddfa0 0x5556858e9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858da6d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- 172.21.15.16:0/1004327912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858d8440 0x5556858da190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- 172.21.15.16:0/1004327912 >> 172.21.15.16:0/1004327912 conn(0x5556858d3ab0 msgr2=0x5556858d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- 172.21.15.16:0/1004327912 shutdown_connections 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- 172.21.15.16:0/1004327912 wait complete. 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- start start 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 Processor -- start 2024-04-25T01:27:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858e5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858ddfa0 0x5556858e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5556858f0260 con 0x5556858d8e30 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5556858efb90 con 0x5556858d8440 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5556858f0930 con 0x5556858ddfa0 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f0298176740 10 client.0 _ll_register_callbacks cb 0x7f02840d2000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47486/0 (socket says 172.21.15.16:47486) 2024-04-25T01:27:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.210+0000 7f028e7fc640 1 -- 172.21.15.16:0/2201201525 learned_addr learned my addr 172.21.15.16:0/2201201525 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028effd640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028f7fe640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858ddfa0 0x5556858e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028e7fc640 1 -- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858ddfa0 msgr2=0x5556858e8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028e7fc640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858ddfa0 0x5556858e8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028e7fc640 1 -- 172.21.15.16:0/2201201525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 msgr2=0x5556858e5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028e7fc640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858e5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028e7fc640 1 -- 172.21.15.16:0/2201201525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556858e91e0 con 0x5556858d8e30 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028effd640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028f7fe640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858ddfa0 0x5556858e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f028e7fc640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e5bd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02840975b0 tx=0x7f028409a9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16be8b42148b6ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0272ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0272ff5640 1 -- 172.21.15.16:0/2201201525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840a0960 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0272ff5640 1 -- 172.21.15.16:0/2201201525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0284096ef0 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02840b4070 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0272ff5640 1 -- 172.21.15.16:0/2201201525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02840662b0 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0298176740 10 client.15234 fetch_fsmap learned FSMap version 21 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5556859550c0 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0272ff5640 1 -- 172.21.15.16:0/2201201525 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f028407e080 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0272ff5640 1 -- 172.21.15.16:0/2201201525 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f02840970f0 con 0x5556858d8e30 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.211+0000 7f0298176740 10 client.15234 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0298176740 10 client.15234 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0298176740 20 client.15234 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0298176740 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x555685959900 0x555685956410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0298176740 4 client.15234 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 --> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] -- command(tid 0: {"prefix": "status"}) -- 0x555685843790 con 0x555685959900 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f028effd640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x555685959900 0x555685956410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f028effd640 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x555685959900 0x555685956410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61f133743d0ee83d server_cookie=3b38fbaf47444f47 in_seq=0 out_seq=0 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0272ff5640 10 client.15234 ms_handle_connect on v2:172.21.15.156:6838/1428073868 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0272ff5640 1 -- 172.21.15.16:0/2201201525 <== mds.1 v2:172.21.15.156:6838/1428073868 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555685843790 con 0x555685959900 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0272ff5640 10 client.15234 handle_command_reply: tid=0 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.212+0000 7f0298176740 1 client.15234 shutdown 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 msgr2=0x5556858e5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e5bd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02840975b0 tx=0x7f028409a9c0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 shutdown_connections 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:6838/1428073868,v1:172.21.15.156:6839/1428073868] conn(0x555685959900 0x555685956410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5556858ddfa0 0x5556858e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0272ff5640 0 client.15234 ms_handle_reset on v2:172.21.15.156:6838/1428073868 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5556858d8e30 0x5556858e5bd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 --2- 172.21.15.16:0/2201201525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556858d8440 0x5556858e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 >> 172.21.15.16:0/2201201525 conn(0x5556858d3ab0 msgr2=0x5556858d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 shutdown_connections 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 1 -- 172.21.15.16:0/2201201525 wait complete. 2024-04-25T01:27:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0298176740 20 client.15234 trim_cache size 0 max 16384 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24499, 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 115.155889276, 2024-04-25T01:27:14.202 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 21, 2024-04-25T01:27:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 87, 2024-04-25T01:27:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-04-25T01:27:14.203 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 132.28619557499999 2024-04-25T01:27:14.203 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:27:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02780a2b90 msgr2=0x7f02780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02780a2b90 0x7f02780a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x5556858d9690 tx=0x7f0288005c50 comp rx=0 tx=0).stop 2024-04-25T01:27:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 msgr2=0x7f0290073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 0x7f0290073b30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f028400d950 tx=0x7f028400de10 comp rx=0 tx=0).stop 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 shutdown_connections 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02780a2b90 0x7f02780a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0290103340 0x7f0290074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0290102980 0x7f0290073b30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 --2- 172.21.15.16:0/1453432276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0290101790 0x7f02900754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 >> 172.21.15.16:0/1453432276 conn(0x7f02900fcd20 msgr2=0x7f0290105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 shutdown_connections 2024-04-25T01:27:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.213+0000 7f0295396640 1 -- 172.21.15.16:0/1453432276 wait complete. 2024-04-25T01:27:14.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:27:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 -- 172.21.15.16:0/2420209498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 msgr2=0x7f4d90101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 --2- 172.21.15.16:0/2420209498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90101bc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009900 tx=0x7f4d8002a410 comp rx=0 tx=0).stop 2024-04-25T01:27:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 -- 172.21.15.16:0/2420209498 shutdown_connections 2024-04-25T01:27:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 --2- 172.21.15.16:0/2420209498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d90103350 0x7f4d90109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 --2- 172.21.15.16:0/2420209498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d90102990 0x7f4d90102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 --2- 172.21.15.16:0/2420209498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90101bc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 -- 172.21.15.16:0/2420209498 >> 172.21.15.16:0/2420209498 conn(0x7f4d900fcd10 msgr2=0x7f4d900ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 -- 172.21.15.16:0/2420209498 shutdown_connections 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.331+0000 7f4d9691d640 1 -- 172.21.15.16:0/2420209498 wait complete. 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 Processor -- start 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 -- start start 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d90102990 0x7f4d90197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d90103350 0x7f4d9019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9010d2a0 con 0x7f4d901017a0 2024-04-25T01:27:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9010d120 con 0x7f4d90102990 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d9691d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9010d420 con 0x7f4d90103350 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d8ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d8ffff640 1 -- 172.21.15.16:0/4160309581 learned_addr learned my addr 172.21.15.16:0/4160309581 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.332+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d90102990 0x7f4d90197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d94eba640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d90103350 0x7f4d9019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8ffff640 1 -- 172.21.15.16:0/4160309581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d90103350 msgr2=0x7f4d9019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d90103350 0x7f4d9019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8ffff640 1 -- 172.21.15.16:0/4160309581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d90102990 msgr2=0x7f4d90197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d90102990 0x7f4d90197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8ffff640 1 -- 172.21.15.16:0/4160309581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9019f620 con 0x7f4d901017a0 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90197a00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4d800314e0 tx=0x7f4d800042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8d7fa640 1 -- 172.21.15.16:0/4160309581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8003f030 con 0x7f4d901017a0 2024-04-25T01:27:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8d7fa640 1 -- 172.21.15.16:0/4160309581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d80003cb0 con 0x7f4d901017a0 2024-04-25T01:27:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d9019f8b0 con 0x7f4d901017a0 2024-04-25T01:27:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d8d7fa640 1 -- 172.21.15.16:0/4160309581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80035070 con 0x7f4d901017a0 2024-04-25T01:27:14.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.333+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d9019fda0 con 0x7f4d901017a0 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.335+0000 7f4d8d7fa640 1 -- 172.21.15.16:0/4160309581 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4d80042030 con 0x7f4d901017a0 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.335+0000 7f4d99622740 1 Processor -- start 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.335+0000 7f4d99622740 1 -- start start 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d99622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0b1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d99622740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0b6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d99622740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d99622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee014f00 con 0x55c0ee0b1440 2024-04-25T01:27:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d99622740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c0edfbd750 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d99622740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee010ec0 con 0x55c0ee0b1e30 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47504/0 (socket says 172.21.15.16:47504) 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 -- 172.21.15.16:0/2644987360 learned_addr learned my addr 172.21.15.16:0/2644987360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 -- 172.21.15.16:0/2644987360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1e30 msgr2=0x55c0ee0b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 -- 172.21.15.16:0/2644987360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b6fa0 msgr2=0x55c0ee0b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0b9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 -- 172.21.15.16:0/2644987360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0edd292d0 con 0x55c0ee0b1440 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d8d7fa640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4d780a2b90 0x7f4d780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4d780a2b90 0x7f4d780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d94eba640 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0b1860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008dc0 tx=0x7f4d8800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e687707df1c3337 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d8d7fa640 1 -- 172.21.15.16:0/4160309581 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4d800ef6a0 con 0x7f4d901017a0 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.336+0000 7f4d767fc640 1 -- 172.21.15.16:0/2644987360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88014070 con 0x55c0ee0b1440 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d767fc640 1 -- 172.21.15.16:0/2644987360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d88003b70 con 0x55c0ee0b1440 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d767fc640 1 -- 172.21.15.16:0/2644987360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88010720 con 0x55c0ee0b1440 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 -- 172.21.15.16:0/2644987360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 msgr2=0x55c0ee0b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d8ffff640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4d780a2b90 0x7f4d780a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4d800ed4f0 tx=0x7f4d800f1040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0b1860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008dc0 tx=0x7f4d8800e720 comp rx=0 tx=0).stop 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 -- 172.21.15.16:0/2644987360 shutdown_connections 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0b9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 --2- 172.21.15.16:0/2644987360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0b1860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 -- 172.21.15.16:0/2644987360 >> 172.21.15.16:0/2644987360 conn(0x55c0ee0acab0 msgr2=0x55c0ee0aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 -- 172.21.15.16:0/2644987360 shutdown_connections 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 -- 172.21.15.16:0/2644987360 wait complete. 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.337+0000 7f4d99622740 1 -- start start 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 Processor -- start 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0b9f50 con 0x55c0ee0b1e30 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c0edfd0cf0 con 0x55c0ee0b1440 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0ba0d0 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d99622740 10 client.0 _ll_register_callbacks cb 0x55c0edfd06e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47518/0 (socket says 172.21.15.16:47518) 2024-04-25T01:27:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 -- 172.21.15.16:0/44045989 learned_addr learned my addr 172.21.15.16:0/44045989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d94eba640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d94eba640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 -- 172.21.15.16:0/44045989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 msgr2=0x55c0ee0c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 -- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1440 msgr2=0x55c0ee0bb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d8f7fe640 1 -- 172.21.15.16:0/44045989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0eddf36b0 con 0x55c0ee0b1e30 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.338+0000 7f4d94eba640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bbe30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400bad0 tx=0x7f4d8400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dab4d943c37ca83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/44045989 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84003dc0 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/44045989 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4d84011440 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d88003b70 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/44045989 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4d84005020 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c0ee0c5e90 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/44045989 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d88003b70 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/44045989 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4d84011680 con 0x55c0ee0b1e30 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 10 client.15246 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 20 client.15246 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee0c85b0 0x55c0ee0c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 4 client.15246 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.339+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0ee022440 con 0x55c0ee0c85b0 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.340+0000 7f4d94eba640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee0c85b0 0x55c0ee0c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.340+0000 7f4d6f7fe640 10 client.15246 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.340+0000 7f4d94eba640 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee0c85b0 0x55c0ee0c9a20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=759ff49ada6b0f0 server_cookie=67148a654ec3622c in_seq=0 out_seq=0 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/44045989 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0ee022440 con 0x55c0ee0c85b0 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d6f7fe640 10 client.15246 handle_command_reply: tid=0 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 client.15246 shutdown 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 msgr2=0x55c0ee0bbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bbe30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400bad0 tx=0x7f4d8400bf90 comp rx=0 tx=0).stop 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 shutdown_connections 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bbe30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee0c85b0 0x55c0ee0c9a20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 --2- 172.21.15.16:0/44045989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d6f7fe640 0 client.15246 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 >> 172.21.15.16:0/44045989 conn(0x55c0ee0acab0 msgr2=0x55c0ee0abae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 shutdown_connections 2024-04-25T01:27:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 1 -- 172.21.15.16:0/44045989 wait complete. 2024-04-25T01:27:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.341+0000 7f4d99622740 20 client.15246 trim_cache size 0 max 16384 2024-04-25T01:27:14.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.359+0000 7f4d99622740 1 Processor -- start 2024-04-25T01:27:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.359+0000 7f4d99622740 1 -- start start 2024-04-25T01:27:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.359+0000 7f4d99622740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.359+0000 7f4d99622740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.359+0000 7f4d99622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.359+0000 7f4d99622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0c7d60 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d99622740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0c8430 con 0x55c0ee0b1e30 2024-04-25T01:27:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d99622740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0c8b00 con 0x55c0ee0b1440 2024-04-25T01:27:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0bba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47520/0 (socket says 172.21.15.16:47520) 2024-04-25T01:27:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 -- 172.21.15.16:0/4100181919 learned_addr learned my addr 172.21.15.16:0/4100181919 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 -- 172.21.15.16:0/4100181919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1440 msgr2=0x55c0ee0bb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 -- 172.21.15.16:0/4100181919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 msgr2=0x55c0ee0c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 -- 172.21.15.16:0/4100181919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0ee0ad780 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d94eba640 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0bba40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008b50 tx=0x7f4d88005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70c8a6b5477ce87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d75ffb640 1 -- 172.21.15.16:0/4100181919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88014040 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d75ffb640 1 -- 172.21.15.16:0/4100181919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d880288f0 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.360+0000 7f4d75ffb640 1 -- 172.21.15.16:0/4100181919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88010c70 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- 172.21.15.16:0/4100181919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 msgr2=0x55c0ee0bba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0bba40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008b50 tx=0x7f4d88005bd0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- 172.21.15.16:0/4100181919 shutdown_connections 2024-04-25T01:27:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0bba40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- 172.21.15.16:0/4100181919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- 172.21.15.16:0/4100181919 >> 172.21.15.16:0/4100181919 conn(0x55c0ee0acab0 msgr2=0x55c0ee0b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- 172.21.15.16:0/4100181919 shutdown_connections 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- 172.21.15.16:0/4100181919 wait complete. 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- start start 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 Processor -- start 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0c98a0 con 0x55c0ee0b1440 2024-04-25T01:27:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0c91d0 con 0x55c0ee0b1e30 2024-04-25T01:27:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.361+0000 7f4d99622740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c0ee0c9f70 con 0x55c0ee0b6fa0 2024-04-25T01:27:14.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d99622740 10 client.0 _ll_register_callbacks cb 0x7f4d840043c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:14.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47528/0 (socket says 172.21.15.16:47528) 2024-04-25T01:27:14.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 -- 172.21.15.16:0/1593191588 learned_addr learned my addr 172.21.15.16:0/1593191588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:14.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 -- 172.21.15.16:0/1593191588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 msgr2=0x55c0ee0c22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 -- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 msgr2=0x55c0ee0bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 -- 172.21.15.16:0/1593191588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0ee0aaf70 con 0x55c0ee0b1440 2024-04-25T01:27:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d94eba640 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bedb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008d90 tx=0x7f4d88008920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1650bdddc47fd4b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:14.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/1593191588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88014050 con 0x55c0ee0b1440 2024-04-25T01:27:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/1593191588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4d88028440 con 0x55c0ee0b1440 2024-04-25T01:27:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d880288f0 con 0x55c0ee0b1440 2024-04-25T01:27:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/1593191588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4d88040050 con 0x55c0ee0b1440 2024-04-25T01:27:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d99622740 10 client.15252 fetch_fsmap learned FSMap version 21 2024-04-25T01:27:14.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c0ee12eb80 con 0x55c0ee0b1440 2024-04-25T01:27:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.362+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/1593191588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d880288f0 con 0x55c0ee0b1440 2024-04-25T01:27:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/1593191588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4d88039830 con 0x55c0ee0b1440 2024-04-25T01:27:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d99622740 10 client.15252 fetch_fsmap finished waiting for FSMap version 21 2024-04-25T01:27:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d99622740 10 client.15252 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:27:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d99622740 20 client.15252 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:14.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d99622740 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee132fd0 0x55c0ee131ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:14.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d99622740 4 client.15252 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:27:14.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "status"}) -- 0x55c0ee01c790 con 0x55c0ee132fd0 2024-04-25T01:27:14.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee132fd0 0x55c0ee131ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:14.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d6f7fe640 10 client.15252 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:14.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.363+0000 7f4d8f7fe640 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee132fd0 0x55c0ee131ce0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31e28e1cda36a1da server_cookie=fb46ae124d859cbb in_seq=0 out_seq=0 2024-04-25T01:27:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d6f7fe640 1 -- 172.21.15.16:0/1593191588 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55c0ee01c790 con 0x55c0ee132fd0 2024-04-25T01:27:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d6f7fe640 10 client.15252 handle_command_reply: tid=0 2024-04-25T01:27:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 client.15252 shutdown 2024-04-25T01:27:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 msgr2=0x55c0ee0bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bedb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008d90 tx=0x7f4d88008920 comp rx=0 tx=0).stop 2024-04-25T01:27:14.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 shutdown_connections 2024-04-25T01:27:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0ee0b6fa0 0x55c0ee0c22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0ee132fd0 0x55c0ee131ce0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0ee0b1e30 0x55c0ee0bf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 --2- 172.21.15.16:0/1593191588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0ee0b1440 0x55c0ee0bedb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d6f7fe640 0 client.15252 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 >> 172.21.15.16:0/1593191588 conn(0x55c0ee0acab0 msgr2=0x55c0ee0bb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 shutdown_connections 2024-04-25T01:27:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 1 -- 172.21.15.16:0/1593191588 wait complete. 2024-04-25T01:27:14.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d99622740 20 client.15252 trim_cache size 0 max 16384 2024-04-25T01:27:14.366 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:27:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "605c42f8-02a1-11ef-bc93-c7b262605968", 2024-04-25T01:27:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-25T01:27:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24524, 2024-04-25T01:27:14.366 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-25T01:27:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-25T01:27:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-25T01:27:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 117.323338952, 2024-04-25T01:27:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 21, 2024-04-25T01:27:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 87, 2024-04-25T01:27:14.367 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2024-04-25T01:27:14.368 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 123.593560282 2024-04-25T01:27:14.368 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:27:14.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4d780a2b90 msgr2=0x7f4d780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d9691d640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4d780a2b90 0x7f4d780a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4d800ed4f0 tx=0x7f4d800f1040 comp rx=0 tx=0).stop 2024-04-25T01:27:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 msgr2=0x7f4d90197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.364+0000 7f4d9691d640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90197a00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4d800314e0 tx=0x7f4d800042e0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 shutdown_connections 2024-04-25T01:27:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4d780a2b90 0x7f4d780a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d90103350 0x7f4d9019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d90102990 0x7f4d90197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 --2- 172.21.15.16:0/4160309581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d901017a0 0x7f4d90197a00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 >> 172.21.15.16:0/4160309581 conn(0x7f4d900fcd10 msgr2=0x7f4d90105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 shutdown_connections 2024-04-25T01:27:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:14.365+0000 7f4d9691d640 1 -- 172.21.15.16:0/4160309581 wait complete. 2024-04-25T01:27:14.373 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:27:14.388 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:27:14.388 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:27:14.389 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:27:14.389 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T01:27:14.423 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T01:27:14.423 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-25T01:27:14.423 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-04-25T01:27:14.423 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:27:14.423 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:14.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:14.424 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-25T01:27:14.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:14.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-25T01:27:14.506 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2024-04-25T01:27:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:27:14.508 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T01:27:14.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:27:14.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:14 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1636303042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:14.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:14 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2626905429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:27:14.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:14 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1636303042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:14.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:14 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2626905429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:14.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:14 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1636303042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:27:14.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:14 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2626905429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:27:15.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:15 smithi156 ceph-mon[32631]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:15.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:15 smithi016 ceph-mon[29153]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:15.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:15 smithi038 ceph-mon[32841]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:27:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.645+0000 7f663d808700 1 -- 172.21.15.16:0/2919450761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101160 msgr2=0x7f66381015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.645+0000 7f663d808700 1 --2- 172.21.15.16:0/2919450761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101160 0x7f66381015e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f662c009990 tx=0x7f662c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 -- 172.21.15.16:0/2919450761 shutdown_connections 2024-04-25T01:27:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 --2- 172.21.15.16:0/2919450761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6638101b20 0x7f663810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 --2- 172.21.15.16:0/2919450761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66381071e0 0x7f66381075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 --2- 172.21.15.16:0/2919450761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101160 0x7f66381015e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 -- 172.21.15.16:0/2919450761 >> 172.21.15.16:0/2919450761 conn(0x7f66380771a0 msgr2=0x7f66380775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 -- 172.21.15.16:0/2919450761 shutdown_connections 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 -- 172.21.15.16:0/2919450761 wait complete. 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 Processor -- start 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.646+0000 7f663d808700 1 -- start start 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6638101160 0x7f663819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 0x7f663819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:16.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381071e0 0x7f66381988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66381113b0 con 0x7f6638101b20 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6638111230 con 0x7f66381071e0 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6638111530 con 0x7f6638101160 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 0x7f663819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 0x7f663819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47540/0 (socket says 172.21.15.16:47540) 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 -- 172.21.15.16:0/1355328434 learned_addr learned my addr 172.21.15.16:0/1355328434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66377fe700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381071e0 0x7f66381988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f6636ffd700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6638101160 0x7f663819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 -- 172.21.15.16:0/1355328434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6638101160 msgr2=0x7f663819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6638101160 0x7f663819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 -- 172.21.15.16:0/1355328434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381071e0 msgr2=0x7f66381988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381071e0 0x7f66381988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 -- 172.21.15.16:0/1355328434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6638199020 con 0x7f6638101b20 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66377fe700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381071e0 0x7f66381988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f66367fc700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 0x7f663819ebf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f662c000c00 tx=0x7f662c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c027070 con 0x7f6638101b20 2024-04-25T01:27:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66381992b0 con 0x7f6638101b20 2024-04-25T01:27:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f662c004140 con 0x7f6638101b20 2024-04-25T01:27:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.647+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c00f460 con 0x7f6638101b20 2024-04-25T01:27:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.648+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66381a5580 con 0x7f6638101b20 2024-04-25T01:27:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.648+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6638102850 con 0x7f6638101b20 2024-04-25T01:27:16.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.650+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f662c02a070 con 0x7f6638101b20 2024-04-25T01:27:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.650+0000 7f663c806700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66200a2c20 0x7f66200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.650+0000 7f6636ffd700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66200a2c20 0x7f66200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.650+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f662c0d0560 con 0x7f6638101b20 2024-04-25T01:27:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.651+0000 7f6636ffd700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66200a2c20 0x7f66200a50e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6628006fd0 tx=0x7f6628008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.654+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f662c031070 con 0x7f6638101b20 2024-04-25T01:27:16.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.914+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6638068f20 con 0x7f6638101b20 2024-04-25T01:27:16.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.914+0000 7f663c806700 1 -- 172.21.15.16:0/1355328434 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7f662c013070 con 0x7f6638101b20 2024-04-25T01:27:16.894 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.916+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66200a2c20 msgr2=0x7f66200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.916+0000 7f663d808700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66200a2c20 0x7f66200a50e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6628006fd0 tx=0x7f6628008040 comp rx=0 tx=0).stop 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.916+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 msgr2=0x7f663819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.916+0000 7f663d808700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 0x7f663819ebf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f662c000c00 tx=0x7f662c0047b0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 shutdown_connections 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f66200a2c20 0x7f66200a50e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381071e0 0x7f66381988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6638101160 0x7f663819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 --2- 172.21.15.16:0/1355328434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6638101b20 0x7f663819ebf0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 >> 172.21.15.16:0/1355328434 conn(0x7f66380771a0 msgr2=0x7f66381056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 shutdown_connections 2024-04-25T01:27:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:16.917+0000 7f663d808700 1 -- 172.21.15.16:0/1355328434 wait complete. 2024-04-25T01:27:17.597 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:17 smithi016 ceph-mon[29153]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:27:17.597 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:17 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1355328434' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:27:17.631 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:27:17.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:17.632 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-25T01:27:17.650 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:27:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:27:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:27:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:27:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-25T01:27:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42093sec preferred_lft 42093sec 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:27:17.651 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:27:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:27:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-25T01:27:17.652 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:27:17.652 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:27:17.652 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-25T01:27:17.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:27:17.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-25T01:27:17.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-25T01:27:17.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:27:17.653 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-25T01:27:17.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:27:17.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:17 smithi156 ceph-mon[32631]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:27:17.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:17 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1355328434' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:27:17.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:27:17.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:17.840 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:27:17.866 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-25T01:27:17.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:17.868 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-25T01:27:17.938 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-25T01:27:17.938 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:27:17.938 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T01:27:17.938 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T01:27:17.938 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:27:17.939 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:27:17.939 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-25T01:27:17.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T01:27:17.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T01:27:17.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T01:27:17.939 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-25T01:27:17.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:17 smithi038 ceph-mon[32841]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:27:17.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:17 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1355328434' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:27:18.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:27:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:27:18.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:18.214 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-25T01:27:18.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:18.230 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-25T01:27:18.285 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:27:18.285 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-25T01:27:18.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:18.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T01:27:18.285 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-25T01:27:18.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:27:18.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:27:18.405 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T01:27:18.405 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-25T01:27:18.455 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:27:18.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:27:18.539 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:27:18.541 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-25T01:27:18.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-25T01:27:18.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:27:18.541 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-25T01:27:18.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:27:18.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:27:18.619 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-04-25T01:27:18.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:27:18.619 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:18.635 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:27:18.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:27:18.635 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:18.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-25T01:27:18.755 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-25T01:27:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-25T01:27:18.812 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/16c0d07067a58c56f45ec23b622751e5261a2e837fe5fa0d543316596f13e381/merged 2024-04-25T01:27:18.813 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d518a07434cef2ba6d0fb38a3375bd1928dc01c73e8e9f73f9c15f69aa9854eb/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/eacd6b09ef4a6e4ba911499c6dda3d8c00f85d92a4beeb18ee3f6385963acddc/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5e57d528aac2e45e6b57eb864f9cdcbb35ca619bdf069d68977ec32cd573d5d8/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d8d83fda482788210156d3f92054280bb69f00c804ff5e126b9d3a48a26cae88/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/6c8a4720e09cdae186afc933b7ba22d798fe84e97451c6732372924dafab379f/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c5837d26ef3baf8d52c157485014fafff794a6e921a981e76d2ed6566ec77ab1/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d2f84f0dd39a6c469d2183d1a7f8b3a4ac6d21dc991cfeaa06d01554b9175cea/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/8ce51f737b75708ced448d9059b575f0d95f7b133c8553a75ddd13dda0841593/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d33513c1448543575adb36929bee42337b46ac20e4231b0cc3a1754128f12908/merged 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-25T01:27:18.814 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:18.815 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:18.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:18.815 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-25T01:27:18.866 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T01:27:18.866 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/32ff4df1-a9e4-48d1-b4f8-1df2db566f01 --client_fs=cephfs) 2024-04-25T01:27:18.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-25T01:27:18.934 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:27:18.990 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-25T01:27:18.990 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-25T01:27:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/16c0d07067a58c56f45ec23b622751e5261a2e837fe5fa0d543316596f13e381/merged 2024-04-25T01:27:18.992 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d518a07434cef2ba6d0fb38a3375bd1928dc01c73e8e9f73f9c15f69aa9854eb/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/eacd6b09ef4a6e4ba911499c6dda3d8c00f85d92a4beeb18ee3f6385963acddc/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5e57d528aac2e45e6b57eb864f9cdcbb35ca619bdf069d68977ec32cd573d5d8/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d8d83fda482788210156d3f92054280bb69f00c804ff5e126b9d3a48a26cae88/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/6c8a4720e09cdae186afc933b7ba22d798fe84e97451c6732372924dafab379f/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c5837d26ef3baf8d52c157485014fafff794a6e921a981e76d2ed6566ec77ab1/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d2f84f0dd39a6c469d2183d1a7f8b3a4ac6d21dc991cfeaa06d01554b9175cea/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/8ce51f737b75708ced448d9059b575f0d95f7b133c8553a75ddd13dda0841593/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d33513c1448543575adb36929bee42337b46ac20e4231b0cc3a1754128f12908/merged 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:18.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:18.994 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-25T01:27:19.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-04-25T01:27:19.027+0000 7f025994e480 -1 init, newargv = 0x560b6f5da020 newargc=15 2024-04-25T01:27:19.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[80756]: starting ceph client 2024-04-25T01:27:19.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[80756]: starting fuse 2024-04-25T01:27:19.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:19 smithi156 ceph-mon[32631]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:19.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:19 smithi016 ceph-mon[29153]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:19.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:19 smithi038 ceph-mon[32841]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:27:20.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-25T01:27:20.156 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:27:20.215 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-25T01:27:20.215 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-25T01:27:20.215 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-25T01:27:20.215 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-25T01:27:20.215 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:27:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/16c0d07067a58c56f45ec23b622751e5261a2e837fe5fa0d543316596f13e381/merged 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d518a07434cef2ba6d0fb38a3375bd1928dc01c73e8e9f73f9c15f69aa9854eb/merged 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/eacd6b09ef4a6e4ba911499c6dda3d8c00f85d92a4beeb18ee3f6385963acddc/merged 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/5e57d528aac2e45e6b57eb864f9cdcbb35ca619bdf069d68977ec32cd573d5d8/merged 2024-04-25T01:27:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d8d83fda482788210156d3f92054280bb69f00c804ff5e126b9d3a48a26cae88/merged 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/6c8a4720e09cdae186afc933b7ba22d798fe84e97451c6732372924dafab379f/merged 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c5837d26ef3baf8d52c157485014fafff794a6e921a981e76d2ed6566ec77ab1/merged 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d2f84f0dd39a6c469d2183d1a7f8b3a4ac6d21dc991cfeaa06d01554b9175cea/merged 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/8ce51f737b75708ced448d9059b575f0d95f7b133c8553a75ddd13dda0841593/merged 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d33513c1448543575adb36929bee42337b46ac20e4231b0cc3a1754128f12908/merged 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:20.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:20.218 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-25T01:27:20.274 INFO:teuthology.orchestra.run.smithi016.stdout:115 2024-04-25T01:27:20.274 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T01:27:20.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-25T01:27:20.275 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-25T01:27:20.276 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-25T01:27:20.277 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T01:27:20.277 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-25T01:27:20.277 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-04-25T01:27:20.277 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-25T01:27:20.402 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.80756.asok 2024-04-25T01:27:20.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:27:20.413 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.80756.asok 2024-04-25T01:27:20.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:20.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.80756.asok status 2024-04-25T01:27:20.553 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:27:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "80756", 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/volumes/qa/sv_0/32ff4df1-a9e4-48d1-b4f8-1df2db566f01" 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:27:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 15264, 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 15264 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-25T01:27:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3938078329 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3938078329 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.15264 192.168.0.1:0/3938078329", 2024-04-25T01:27:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3938078329", 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 5, 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 21, 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 87, 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-25T01:27:20.557 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:27:20.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:20.563 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-25T01:27:20.563 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:20.582 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-25T01:27:20.582 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:20.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:27:20.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:20.648 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:27:20.658 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:27:20.658 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:27:20.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-25T01:27:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 -- 172.21.15.16:0/134338875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308100750 msgr2=0x7fa308100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/134338875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308100750 0x7fa308100b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa304009900 tx=0x7fa30402a410 comp rx=0 tx=0).stop 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 -- 172.21.15.16:0/134338875 shutdown_connections 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/134338875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa308102300 0x7fa308108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/134338875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa308101940 0x7fa308101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/134338875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308100750 0x7fa308100b70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 -- 172.21.15.16:0/134338875 >> 172.21.15.16:0/134338875 conn(0x7fa3080fbcc0 msgr2=0x7fa3080fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 -- 172.21.15.16:0/134338875 shutdown_connections 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.812+0000 7fa30f8bd640 1 -- 172.21.15.16:0/134338875 wait complete. 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 Processor -- start 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 -- start start 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa308100750 0x7fa30806aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa308101940 0x7fa30819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 0x7fa30819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:20.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa30810c260 con 0x7fa308102300 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa30810c0e0 con 0x7fa308100750 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30f8bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa30810c3e0 con 0x7fa308101940 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 0x7fa30819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 0x7fa30819ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47570/0 (socket says 172.21.15.16:47570) 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 -- 172.21.15.16:0/749941630 learned_addr learned my addr 172.21.15.16:0/749941630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30d659640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa308100750 0x7fa30806aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30ce58640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa308101940 0x7fa30819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 -- 172.21.15.16:0/749941630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa308101940 msgr2=0x7fa30819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa308101940 0x7fa30819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 -- 172.21.15.16:0/749941630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa308100750 msgr2=0x7fa30806aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa308100750 0x7fa30806aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.813+0000 7fa30de5a640 1 -- 172.21.15.16:0/749941630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30819f620 con 0x7fa308102300 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.814+0000 7fa30de5a640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 0x7fa30819ed10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa30000b970 tx=0x7fa30000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.814+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300019070 con 0x7fa308102300 2024-04-25T01:27:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.814+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30819f8b0 con 0x7fa308102300 2024-04-25T01:27:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.814+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa300014440 con 0x7fa308102300 2024-04-25T01:27:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.814+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3000075d0 con 0x7fa308102300 2024-04-25T01:27:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.814+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30819fe80 con 0x7fa308102300 2024-04-25T01:27:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.815+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa308107a10 con 0x7fa308102300 2024-04-25T01:27:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.816+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa30001c070 con 0x7fa308102300 2024-04-25T01:27:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.817+0000 7fa2fe7fc640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2f40a2c30 0x7fa2f40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.817+0000 7fa30d659640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2f40a2c30 0x7fa2f40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.817+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa3000ce640 con 0x7fa308102300 2024-04-25T01:27:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.817+0000 7fa30d659640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2f40a2c30 0x7fa2f40a50f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa304004770 tx=0x7fa304000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:20.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:20.821+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa30009af30 con 0x7fa308102300 2024-04-25T01:27:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.047+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa308062340 con 0x7fa308102300 2024-04-25T01:27:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.504+0000 7fa2fe7fc640 1 -- 172.21.15.16:0/749941630 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7fa30009f230 con 0x7fa308102300 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2f40a2c30 msgr2=0x7fa2f40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2f40a2c30 0x7fa2f40a50f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa304004770 tx=0x7fa304000bc0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 msgr2=0x7fa30819ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 0x7fa30819ed10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa30000b970 tx=0x7fa30000be30 comp rx=0 tx=0).stop 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 shutdown_connections 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa2f40a2c30 0x7fa2f40a50f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa308102300 0x7fa30819ed10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa308101940 0x7fa30819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 --2- 172.21.15.16:0/749941630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa308100750 0x7fa30806aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 >> 172.21.15.16:0/749941630 conn(0x7fa3080fbcc0 msgr2=0x7fa308104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 shutdown_connections 2024-04-25T01:27:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.507+0000 7fa30f8bd640 1 -- 172.21.15.16:0/749941630 wait complete. 2024-04-25T01:27:21.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-25T01:27:21.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:21 smithi016 ceph-mon[29153]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:27:21.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:21 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/749941630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T01:27:21.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:21 smithi156 ceph-mon[32631]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:27:21.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:21 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/749941630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.860+0000 7f4aba1ee640 1 -- 172.21.15.16:0/281021465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4101940 msgr2=0x7f4ab4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.860+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/281021465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4101940 0x7f4ab4101dc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009900 tx=0x7f4aa802a410 comp rx=0 tx=0).stop 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 -- 172.21.15.16:0/281021465 shutdown_connections 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/281021465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 0x7f4ab4108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/281021465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4101940 0x7f4ab4101dc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/281021465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ab4100750 0x7f4ab4100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 -- 172.21.15.16:0/281021465 >> 172.21.15.16:0/281021465 conn(0x7f4ab40fbcc0 msgr2=0x7f4ab40fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:21.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 -- 172.21.15.16:0/281021465 shutdown_connections 2024-04-25T01:27:21.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 -- 172.21.15.16:0/281021465 wait complete. 2024-04-25T01:27:21.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 Processor -- start 2024-04-25T01:27:21.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.861+0000 7f4aba1ee640 1 -- start start 2024-04-25T01:27:21.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 0x7f4ab41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:21.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ab4101940 0x7f4ab4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 0x7f4ab419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab410c210 con 0x7f4ab4100750 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab410c090 con 0x7f4ab4102300 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab410c390 con 0x7f4ab4101940 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 0x7f4ab41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 0x7f4ab41979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38504/0 (socket says 172.21.15.16:38504) 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 -- 172.21.15.16:0/2204543889 learned_addr learned my addr 172.21.15.16:0/2204543889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab3fff640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 0x7f4ab419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab2ffd640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ab4101940 0x7f4ab4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:21.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 -- 172.21.15.16:0/2204543889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ab4101940 msgr2=0x7f4ab4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ab4101940 0x7f4ab4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 -- 172.21.15.16:0/2204543889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 msgr2=0x7f4ab419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 0x7f4ab419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 -- 172.21.15.16:0/2204543889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab419f570 con 0x7f4ab4100750 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab3fff640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 0x7f4ab419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab37fe640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 0x7f4ab41979a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400e9c0 tx=0x7f4aa400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa400ccc0 con 0x7f4ab4100750 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab419f860 con 0x7f4ab4100750 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4aa40035f0 con 0x7f4ab4100750 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.862+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa400ce60 con 0x7f4ab4100750 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.863+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab419fd30 con 0x7f4ab4100750 2024-04-25T01:27:21.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.864+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab419fff0 con 0x7f4ab4100750 2024-04-25T01:27:21.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.865+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4aa40037a0 con 0x7f4ab4100750 2024-04-25T01:27:21.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.865+0000 7f4ab0ff9640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4a9c0a2c30 0x7f4a9c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:21.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.865+0000 7f4ab2ffd640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4a9c0a2c30 0x7f4a9c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:21.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.865+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4aa40cf560 con 0x7f4ab4100750 2024-04-25T01:27:21.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.866+0000 7f4ab2ffd640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4a9c0a2c30 0x7f4a9c0a50f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006010 tx=0x7f4aa8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:21.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:21.870+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4aa409c4a0 con 0x7f4ab4100750 2024-04-25T01:27:21.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:21 smithi038 ceph-mon[32841]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:27:21.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:21 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/749941630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T01:27:22.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.086+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 --> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f4ab40fde70 con 0x7f4a9c0a2c30 2024-04-25T01:27:22.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.107+0000 7f4ab0ff9640 1 -- 172.21.15.16:0/2204543889 <== mgr.14152 v2:172.21.15.16:6800/1051975299 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f4ab40fde70 con 0x7f4a9c0a2c30 2024-04-25T01:27:22.086 INFO:teuthology.orchestra.run.smithi016.stdout:{} 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4a9c0a2c30 msgr2=0x7f4a9c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4a9c0a2c30 0x7f4a9c0a50f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006010 tx=0x7f4aa8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 msgr2=0x7f4ab41979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 0x7f4ab41979a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400e9c0 tx=0x7f4aa400edc0 comp rx=0 tx=0).stop 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 shutdown_connections 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4a9c0a2c30 0x7f4a9c0a50f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ab4102300 0x7f4ab419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ab4101940 0x7f4ab4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 --2- 172.21.15.16:0/2204543889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab4100750 0x7f4ab41979a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 >> 172.21.15.16:0/2204543889 conn(0x7f4ab40fbcc0 msgr2=0x7f4ab40fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 shutdown_connections 2024-04-25T01:27:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:27:22.109+0000 7f4aba1ee640 1 -- 172.21.15.16:0/2204543889 wait complete. 2024-04-25T01:27:22.099 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T01:27:22.113 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T01:27:22.126 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:27:22.126 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T01:27:22.142 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T01:27:22.157 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:27:22.157 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T01:27:22.157 INFO:tasks.workunit:timeout=3h 2024-04-25T01:27:22.157 INFO:tasks.workunit:cleanup=True 2024-04-25T01:27:22.158 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:22.174 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 73h/115d Inode: 1 Links: 2 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-25 01:25:21.783891733 +0000 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-25 01:25:21.783891733 +0000 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-25 01:27:20.667367904 +0000 2024-04-25T01:27:22.175 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-04-25T01:27:22.175 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T01:27:22.175 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T01:27:22.246 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:27:22.303 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T01:27:22.360 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:27:22.091+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:27:22.360 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:27:22.091+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:27:22.360 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:27:22.091+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:27:22.360 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:27:22.091+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:27:22.360 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:27:22.091+0000 7f02fd7c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:27:22.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:22 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/749941630' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T01:27:22.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:22 smithi156 ceph-mon[32631]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:27:22.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/749941630' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T01:27:22.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:22 smithi016 ceph-mon[29153]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:27:22.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:22 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/749941630' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T01:27:22.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:22 smithi038 ceph-mon[32841]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:27:23.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:23 smithi016 ceph-mon[29153]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:27:23.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:23 smithi016 ceph-mon[29153]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:23.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:23 smithi038 ceph-mon[32841]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:27:23.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:23 smithi038 ceph-mon[32841]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:24.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:23 smithi156 ceph-mon[32631]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:27:24.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:23 smithi156 ceph-mon[32631]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:25.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:25 smithi016 ceph-mon[29153]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-25T01:27:25.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:25 smithi016 ceph-mon[29153]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:27:25.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:25 smithi038 ceph-mon[32841]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-25T01:27:25.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:25 smithi038 ceph-mon[32841]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:27:26.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:25 smithi156 ceph-mon[32631]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-25T01:27:26.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:25 smithi156 ceph-mon[32631]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:27:27.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:27 smithi016 ceph-mon[29153]: pgmap v360: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-04-25T01:27:27.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:27 smithi038 ceph-mon[32841]: pgmap v360: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-04-25T01:27:28.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:27 smithi156 ceph-mon[32631]: pgmap v360: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-04-25T01:27:29.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:29 smithi016 ceph-mon[29153]: pgmap v361: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-25T01:27:29.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:29 smithi038 ceph-mon[32841]: pgmap v361: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-25T01:27:30.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:29 smithi156 ceph-mon[32631]: pgmap v361: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-25T01:27:31.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:31 smithi016 ceph-mon[29153]: pgmap v362: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:27:31.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:31 smithi038 ceph-mon[32841]: pgmap v362: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:27:32.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:31 smithi156 ceph-mon[32631]: pgmap v362: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:27:33.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:33 smithi016 ceph-mon[29153]: pgmap v363: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-25T01:27:33.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:33 smithi038 ceph-mon[32841]: pgmap v363: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-25T01:27:34.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:33 smithi156 ceph-mon[32631]: pgmap v363: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 -- 172.21.15.16:0/864398384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf41017b0 msgr2=0x7f4bf4101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/864398384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf41017b0 0x7f4bf4101bd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009900 tx=0x7f4bec02a410 comp rx=0 tx=0).stop 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 -- 172.21.15.16:0/864398384 shutdown_connections 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/864398384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf4103360 0x7f4bf4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/864398384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf41029a0 0x7f4bf4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/864398384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf41017b0 0x7f4bf4101bd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.297+0000 7f4bf95fb640 1 -- 172.21.15.16:0/864398384 >> 172.21.15.16:0/864398384 conn(0x7f4bf40fcd00 msgr2=0x7f4bf40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 -- 172.21.15.16:0/864398384 shutdown_connections 2024-04-25T01:27:35.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 -- 172.21.15.16:0/864398384 wait complete. 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 Processor -- start 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 -- start start 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf41017b0 0x7f4bf40754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf41029a0 0x7f4bf4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 0x7f4bf4074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf410d2b0 con 0x7f4bf4103360 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf410d130 con 0x7f4bf41017b0 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.298+0000 7f4bf95fb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf410d430 con 0x7f4bf41029a0 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 0x7f4bf4074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 0x7f4bf4074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45078/0 (socket says 172.21.15.16:45078) 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 -- 172.21.15.16:0/291043123 learned_addr learned my addr 172.21.15.16:0/291043123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf41017b0 0x7f4bf40754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf41029a0 0x7f4bf4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 -- 172.21.15.16:0/291043123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf41029a0 msgr2=0x7f4bf4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf41029a0 0x7f4bf4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 -- 172.21.15.16:0/291043123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf41017b0 msgr2=0x7f4bf40754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf41017b0 0x7f4bf40754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 -- 172.21.15.16:0/291043123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf4074900 con 0x7f4bf4103360 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 0x7f4bf4074070 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009fd0 tx=0x7f4be400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bdbfff640 1 -- 172.21.15.16:0/291043123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4017070 con 0x7f4bf4103360 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bdbfff640 1 -- 172.21.15.16:0/291043123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4be4003e20 con 0x7f4bf4103360 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf41a1a80 con 0x7f4bf4103360 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bdbfff640 1 -- 172.21.15.16:0/291043123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be40106c0 con 0x7f4bf4103360 2024-04-25T01:27:35.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.299+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf41a1f80 con 0x7f4bf4103360 2024-04-25T01:27:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.301+0000 7f4bdbfff640 1 -- 172.21.15.16:0/291043123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f4be4003970 con 0x7f4bf4103360 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 Processor -- start 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 -- start start 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bdbfff640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4bdc0a2cd0 0x7f4bdc0a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61ba8840 0x555c61bad340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 0x555c61bafe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555c61b0bcb0 con 0x555c61bad9b0 2024-04-25T01:27:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c61ab45b0 con 0x555c61ba8840 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555c61b07c70 con 0x555c61ba7e50 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4bdc0a2cd0 0x7f4bdc0a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bdbfff640 1 -- 172.21.15.16:0/291043123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4be40cef50 con 0x7f4bf4103360 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 0x555c61bafe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 0x555c61bafe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45080/0 (socket says 172.21.15.16:45080) 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 -- 172.21.15.16:0/1062711513 learned_addr learned my addr 172.21.15.16:0/1062711513 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61ba8840 0x555c61bad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 -- 172.21.15.16:0/1062711513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 msgr2=0x555c61ba8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 -- 172.21.15.16:0/1062711513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61ba8840 msgr2=0x555c61bad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.302+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61ba8840 0x555c61bad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bf37fe640 1 -- 172.21.15.16:0/1062711513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c61820210 con 0x555c61bad9b0 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4bdc0a2cd0 0x7f4bdc0a5190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4075260 tx=0x7f4be8009f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 0x555c61bafe40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4be4097fa0 tx=0x7f4be401a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f93e566af251b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bd97fa640 1 -- 172.21.15.16:0/1062711513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4020070 con 0x555c61bad9b0 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bd97fa640 1 -- 172.21.15.16:0/1062711513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4be409bcd0 con 0x555c61bad9b0 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bd97fa640 1 -- 172.21.15.16:0/1062711513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be40a4240 con 0x555c61bad9b0 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bfc3df740 1 -- 172.21.15.16:0/1062711513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 msgr2=0x555c61bafe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.303+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 0x555c61bafe40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4be4097fa0 tx=0x7f4be401a360 comp rx=0 tx=0).stop 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- 172.21.15.16:0/1062711513 shutdown_connections 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61bad9b0 0x555c61bafe40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61ba8840 0x555c61bad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/1062711513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- 172.21.15.16:0/1062711513 >> 172.21.15.16:0/1062711513 conn(0x555c61ba34c0 msgr2=0x555c61ba38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- 172.21.15.16:0/1062711513 shutdown_connections 2024-04-25T01:27:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- 172.21.15.16:0/1062711513 wait complete. 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- start start 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 Processor -- start 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555c61bb0960 con 0x555c61ba7e50 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c61ac7b50 con 0x555c61bad9b0 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555c61bb0ae0 con 0x555c61ba8840 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bfc3df740 10 client.0 _ll_register_callbacks cb 0x555c61b02ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bf2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bf2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45086/0 (socket says 172.21.15.16:45086) 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/864812357 learned_addr learned my addr 172.21.15.16:0/864812357 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:35.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.304+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/864812357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 msgr2=0x555c61bb2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 msgr2=0x555c61bb9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/864812357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c618ec620 con 0x555c61ba7e50 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb2310 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4102800 tx=0x7f4bec009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0b54d2e5b03824 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/864812357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec035040 con 0x555c61ba7e50 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/864812357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4bec004240 con 0x555c61ba7e50 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4be409bcd0 con 0x555c61ba7e50 2024-04-25T01:27:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/864812357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4bec04c070 con 0x555c61ba7e50 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bfc3df740 10 client.15294 fetch_fsmap learned FSMap version 22 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555c61bbcc70 con 0x555c61ba7e50 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/864812357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4be409bcd0 con 0x555c61ba7e50 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.305+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/864812357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4bec03c4d0 con 0x555c61ba7e50 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bfc3df740 10 client.15294 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bfc3df740 10 client.15294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bfc3df740 20 client.15294 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61bbe390 0x555c61bc0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bfc3df740 4 client.15294 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c61b191f0 con 0x555c61bbe390 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61bbe390 0x555c61bc0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bd6ff5640 10 client.15294 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.306+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61bbe390 0x555c61bc0820 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3eca8a11cb5e749 server_cookie=4569c2a3af2d3b8f in_seq=0 out_seq=0 2024-04-25T01:27:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/864812357 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c61b191f0 con 0x555c61bbe390 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bd6ff5640 10 client.15294 handle_command_reply: tid=0 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 client.15294 shutdown 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 msgr2=0x555c61bb2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb2310 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4102800 tx=0x7f4bec009670 comp rx=0 tx=0).stop 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 shutdown_connections 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61bbe390 0x555c61bc0820 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bd6ff5640 0 client.15294 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/864812357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb2310 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 >> 172.21.15.16:0/864812357 conn(0x555c61ba34c0 msgr2=0x555c61ba25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 shutdown_connections 2024-04-25T01:27:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 1 -- 172.21.15.16:0/864812357 wait complete. 2024-04-25T01:27:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.307+0000 7f4bfc3df740 20 client.15294 trim_cache size 0 max 16384 2024-04-25T01:27:35.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 Processor -- start 2024-04-25T01:27:35.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 -- start start 2024-04-25T01:27:35.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 0x555c61bb9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bc5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf41ad680 con 0x555c61ba8840 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf41aaf50 con 0x555c61bad9b0 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf41a8da0 con 0x555c61ba7e50 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bf27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 0x555c61bb9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bf27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 0x555c61bb9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45094/0 (socket says 172.21.15.16:45094) 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bf27fc640 1 -- 172.21.15.16:0/2347581629 learned_addr learned my addr 172.21.15.16:0/2347581629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bc5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.322+0000 7f4bf27fc640 1 -- 172.21.15.16:0/2347581629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 msgr2=0x555c61ba6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bf27fc640 1 -- 172.21.15.16:0/2347581629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 msgr2=0x555c61bc5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bc5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bf27fc640 1 -- 172.21.15.16:0/2347581629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c61bc0ea0 con 0x555c61ba8840 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 0x555c61bb9850 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4be800f6e0 tx=0x7f4be8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc3aa6a8ec4c8a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bd8ff9640 1 -- 172.21.15.16:0/2347581629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be80292d0 con 0x555c61ba8840 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bd8ff9640 1 -- 172.21.15.16:0/2347581629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4be802f070 con 0x555c61ba8840 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bd8ff9640 1 -- 172.21.15.16:0/2347581629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8020cc0 con 0x555c61ba8840 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 -- 172.21.15.16:0/2347581629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 msgr2=0x555c61bb9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 0x555c61bb9850 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4be800f6e0 tx=0x7f4be8005f90 comp rx=0 tx=0).stop 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 -- 172.21.15.16:0/2347581629 shutdown_connections 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bc5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba8840 0x555c61bb9850 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/2347581629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba7e50 0x555c61ba6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 -- 172.21.15.16:0/2347581629 >> 172.21.15.16:0/2347581629 conn(0x555c61ba34c0 msgr2=0x555c61ba6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.323+0000 7f4bfc3df740 1 -- 172.21.15.16:0/2347581629 shutdown_connections 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 -- 172.21.15.16:0/2347581629 wait complete. 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 -- start start 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 Processor -- start 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf41a9410 con 0x555c61ba7e50 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf41aa880 con 0x555c61bad9b0 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf41a9ae0 con 0x555c61ba8840 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bfc3df740 10 client.0 _ll_register_callbacks cb 0x7f4bec031c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45110/0 (socket says 172.21.15.16:45110) 2024-04-25T01:27:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/3920459305 learned_addr learned my addr 172.21.15.16:0/3920459305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/3920459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 msgr2=0x555c61bb78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 msgr2=0x555c61bb88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf2ffd640 1 -- 172.21.15.16:0/3920459305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c61bb8ea0 con 0x555c61ba7e50 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.324+0000 7f4bf37fe640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bf2ffd640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb7360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009900 tx=0x7f4bec03cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b88138e575959a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/3920459305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec035040 con 0x555c61ba7e50 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/3920459305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4bec004030 con 0x555c61ba7e50 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4be802f070 con 0x555c61ba7e50 2024-04-25T01:27:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/3920459305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4bec060050 con 0x555c61ba7e50 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 10 client.15300 fetch_fsmap learned FSMap version 22 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555c61c24e60 con 0x555c61ba7e50 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/3920459305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4be802f070 con 0x555c61ba7e50 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/3920459305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4bec003c30 con 0x555c61ba7e50 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 10 client.15300 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 10 client.15300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 20 client.15300 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61c25880 0x555c61c27ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 4 client.15300 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.325+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c61b13540 con 0x555c61c25880 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.326+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61c25880 0x555c61c27ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.326+0000 7f4bf27fc640 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61c25880 0x555c61c27ca0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86be4531d06aaa90 server_cookie=8e2e9d3173ec2232 in_seq=0 out_seq=0 2024-04-25T01:27:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.326+0000 7f4bd6ff5640 10 client.15300 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bd6ff5640 1 -- 172.21.15.16:0/3920459305 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c61b13540 con 0x555c61c25880 2024-04-25T01:27:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bd6ff5640 10 client.15300 handle_command_reply: tid=0 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 client.15300 shutdown 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 msgr2=0x555c61bb7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb7360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009900 tx=0x7f4bec03cbb0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 shutdown_connections 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555c61c25880 0x555c61c27ca0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c61bad9b0 0x555c61bb88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bd6ff5640 0 client.15300 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c61ba8840 0x555c61bb78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 --2- 172.21.15.16:0/3920459305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555c61ba7e50 0x555c61bb7360 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 >> 172.21.15.16:0/3920459305 conn(0x555c61ba34c0 msgr2=0x555c61ba6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 shutdown_connections 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 1 -- 172.21.15.16:0/3920459305 wait complete. 2024-04-25T01:27:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bfc3df740 20 client.15300 trim_cache size 0 max 16384 2024-04-25T01:27:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:27:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:27:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:27:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4bdc0a2cd0 msgr2=0x7f4bdc0a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4bdc0a2cd0 0x7f4bdc0a5190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4075260 tx=0x7f4be8009f90 comp rx=0 tx=0).stop 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 msgr2=0x7f4bf4074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.339+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 0x7f4bf4074070 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009fd0 tx=0x7f4be400b260 comp rx=0 tx=0).stop 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 shutdown_connections 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4bdc0a2cd0 0x7f4bdc0a5190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bf4103360 0x7f4bf4074070 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf41029a0 0x7f4bf4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 --2- 172.21.15.16:0/291043123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf41017b0 0x7f4bf40754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 >> 172.21.15.16:0/291043123 conn(0x7f4bf40fcd00 msgr2=0x7f4bf4105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 shutdown_connections 2024-04-25T01:27:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:35.340+0000 7f4bf95fb640 1 -- 172.21.15.16:0/291043123 wait complete. 2024-04-25T01:27:35.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:35 smithi016 ceph-mon[29153]: pgmap v364: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-25T01:27:35.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:35 smithi038 ceph-mon[32841]: pgmap v364: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-25T01:27:35.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:35 smithi156 ceph-mon[32631]: pgmap v364: 97 pgs: 97 active+clean; 654 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-25T01:27:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:37 smithi016 ceph-mon[29153]: pgmap v365: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:27:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:37 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:37.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:37 smithi038 ceph-mon[32841]: pgmap v365: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:27:37.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:37 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:37.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:37 smithi156 ceph-mon[32631]: pgmap v365: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:27:38.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:37 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:38.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:38 smithi156 ceph-mon[32631]: pgmap v366: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-25T01:27:38.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:38 smithi016 ceph-mon[29153]: pgmap v366: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-25T01:27:38.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:38 smithi038 ceph-mon[32841]: pgmap v366: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-25T01:27:40.465 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.465 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:40.466 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:40.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:27:41.499 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:41 smithi156 ceph-mon[32631]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T01:27:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:41 smithi016 ceph-mon[29153]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T01:27:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:41 smithi038 ceph-mon[32841]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T01:27:43.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:43 smithi156 ceph-mon[32631]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T01:27:43.517 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:43 smithi038 ceph-mon[32841]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T01:27:43.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:43 smithi016 ceph-mon[29153]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T01:27:45.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:45 smithi016 ceph-mon[29153]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-25T01:27:45.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:45 smithi156 ceph-mon[32631]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-25T01:27:45.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:45 smithi038 ceph-mon[32841]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-25T01:27:47.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:47 smithi156 ceph-mon[32631]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T01:27:47.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:47 smithi016 ceph-mon[29153]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T01:27:47.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:47 smithi038 ceph-mon[32841]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 -- 172.21.15.16:0/664508786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa84102250 msgr2=0x7faa84108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 --2- 172.21.15.16:0/664508786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa84102250 0x7faa84108ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009ca0 tx=0x7faa7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 -- 172.21.15.16:0/664508786 shutdown_connections 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 --2- 172.21.15.16:0/664508786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa84102250 0x7faa84108ae0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 --2- 172.21.15.16:0/664508786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa84101890 0x7faa84101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 --2- 172.21.15.16:0/664508786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa841006a0 0x7faa84100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 -- 172.21.15.16:0/664508786 >> 172.21.15.16:0/664508786 conn(0x7faa840fbc10 msgr2=0x7faa840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 -- 172.21.15.16:0/664508786 shutdown_connections 2024-04-25T01:27:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.439+0000 7faa8b363640 1 -- 172.21.15.16:0/664508786 wait complete. 2024-04-25T01:27:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 Processor -- start 2024-04-25T01:27:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 -- start start 2024-04-25T01:27:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 0x7faa841978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa84101890 0x7faa84197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa84102250 0x7faa8419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa8410c1d0 con 0x7faa841006a0 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa8410c050 con 0x7faa84102250 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa8b363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa8410c350 con 0x7faa84101890 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 0x7faa841978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 0x7faa841978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45216/0 (socket says 172.21.15.16:45216) 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 -- 172.21.15.16:0/1221775985 learned_addr learned my addr 172.21.15.16:0/1221775985 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa888fe640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa84101890 0x7faa84197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa89900640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa84102250 0x7faa8419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 -- 172.21.15.16:0/1221775985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa84101890 msgr2=0x7faa84197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa84101890 0x7faa84197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 -- 172.21.15.16:0/1221775985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa84102250 msgr2=0x7faa8419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa84102250 0x7faa8419ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.440+0000 7faa890ff640 1 -- 172.21.15.16:0/1221775985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8419f4f0 con 0x7faa841006a0 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.441+0000 7faa890ff640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 0x7faa841978d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7faa8000d950 tx=0x7faa8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.441+0000 7faa7a7fc640 1 -- 172.21.15.16:0/1221775985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80017070 con 0x7faa841006a0 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.441+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8419f7e0 con 0x7faa841006a0 2024-04-25T01:27:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.441+0000 7faa7a7fc640 1 -- 172.21.15.16:0/1221775985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa80009c30 con 0x7faa841006a0 2024-04-25T01:27:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.441+0000 7faa7a7fc640 1 -- 172.21.15.16:0/1221775985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8000f620 con 0x7faa841006a0 2024-04-25T01:27:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.441+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8419fce0 con 0x7faa841006a0 2024-04-25T01:27:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.443+0000 7faa7a7fc640 1 -- 172.21.15.16:0/1221775985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7faa8001a070 con 0x7faa841006a0 2024-04-25T01:27:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 Processor -- start 2024-04-25T01:27:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 -- start start 2024-04-25T01:27:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x558971768270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x55897176d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5589716cbcb0 con 0x55897176d9b0 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5589716745b0 con 0x558971768840 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa8e073740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5589716c7c70 con 0x558971767e50 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa888fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa7a7fc640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa700a2cd0 0x7faa700a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa888fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45222/0 (socket says 172.21.15.16:45222) 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa888fe640 1 -- 172.21.15.16:0/3715821054 learned_addr learned my addr 172.21.15.16:0/3715821054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa89900640 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x558971768270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa890ff640 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x55897176d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa89900640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa700a2cd0 0x7faa700a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.444+0000 7faa7a7fc640 1 -- 172.21.15.16:0/1221775985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa800cecb0 con 0x7faa841006a0 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa888fe640 1 -- 172.21.15.16:0/3715821054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 msgr2=0x558971768270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa888fe640 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x558971768270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa888fe640 1 -- 172.21.15.16:0/3715821054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 msgr2=0x55897176d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa888fe640 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x55897176d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa888fe640 1 -- 172.21.15.16:0/3715821054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589713e0210 con 0x55897176d9b0 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa890ff640 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x55897176d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa888fe640 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176fe40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7faa7400bc90 tx=0x7faa740107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259fe9043477110c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa89900640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa700a2cd0 0x7faa700a5190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009a60 tx=0x7faa7c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa6f7fe640 1 -- 172.21.15.16:0/3715821054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74005220 con 0x55897176d9b0 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa6f7fe640 1 -- 172.21.15.16:0/3715821054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa7400be40 con 0x55897176d9b0 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.445+0000 7faa6f7fe640 1 -- 172.21.15.16:0/3715821054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa740117b0 con 0x55897176d9b0 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 -- 172.21.15.16:0/3715821054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 msgr2=0x55897176fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176fe40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7faa7400bc90 tx=0x7faa740107a0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 -- 172.21.15.16:0/3715821054 shutdown_connections 2024-04-25T01:27:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176fe40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x55897176d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 --2- 172.21.15.16:0/3715821054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x558971768270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 -- 172.21.15.16:0/3715821054 >> 172.21.15.16:0/3715821054 conn(0x5589717634c0 msgr2=0x5589717638c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 -- 172.21.15.16:0/3715821054 shutdown_connections 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 -- 172.21.15.16:0/3715821054 wait complete. 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 -- start start 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.446+0000 7faa8e073740 1 Processor -- start 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971772270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971768840 0x5589717727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x5589717797f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558971770960 con 0x55897176d9b0 2024-04-25T01:27:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558971687b50 con 0x558971767e50 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558971770ae0 con 0x558971768840 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa8e073740 10 client.0 _ll_register_callbacks cb 0x5589716c2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x5589717797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x5589717797f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45226/0 (socket says 172.21.15.16:45226) 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 -- 172.21.15.16:0/1464402135 learned_addr learned my addr 172.21.15.16:0/1464402135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa888fe640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971768840 0x5589717727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa890ff640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971772270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 -- 172.21.15.16:0/1464402135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971768840 msgr2=0x5589717727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971768840 0x5589717727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 -- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 msgr2=0x558971772270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971772270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 -- 172.21.15.16:0/1464402135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589714ac620 con 0x55897176d9b0 2024-04-25T01:27:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa89900640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x5589717797f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa7c035000 tx=0x7faa7c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c922e3b8fea704e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.447+0000 7faa6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa6cff9640 1 -- 172.21.15.16:0/1464402135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c051070 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa6cff9640 1 -- 172.21.15.16:0/1464402135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa7c0047a0 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5589716d91f0 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa6cff9640 1 -- 172.21.15.16:0/1464402135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faa7c0305c0 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 10 client.15312 fetch_fsmap learned FSMap version 22 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55897177cbd0 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa6cff9640 1 -- 172.21.15.16:0/1464402135 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5589716d91f0 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa6cff9640 1 -- 172.21.15.16:0/1464402135 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faa7c04a040 con 0x55897176d9b0 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 10 client.15312 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 10 client.15312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 20 client.15312 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55897177e2f0 0x558971780780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 4 client.15312 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.448+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589716d3540 con 0x55897177e2f0 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.449+0000 7faa890ff640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55897177e2f0 0x558971780780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.449+0000 7faa890ff640 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55897177e2f0 0x558971780780 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe6d02cb542581b server_cookie=fcf1b5164d8cb199 in_seq=0 out_seq=0 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.449+0000 7faa6cff9640 10 client.15312 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa6cff9640 1 -- 172.21.15.16:0/1464402135 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589716d3540 con 0x55897177e2f0 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa6cff9640 10 client.15312 handle_command_reply: tid=0 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 client.15312 shutdown 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 msgr2=0x5589717797f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x5589717797f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa7c035000 tx=0x7faa7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 shutdown_connections 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x5589717797f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971768840 0x5589717727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55897177e2f0 0x558971780780 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 --2- 172.21.15.16:0/1464402135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971772270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa6cff9640 0 client.15312 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 >> 172.21.15.16:0/1464402135 conn(0x5589717634c0 msgr2=0x5589717625c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 shutdown_connections 2024-04-25T01:27:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 1 -- 172.21.15.16:0/1464402135 wait complete. 2024-04-25T01:27:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.450+0000 7faa8e073740 20 client.15312 trim_cache size 0 max 16384 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.465+0000 7faa8e073740 1 Processor -- start 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.465+0000 7faa8e073740 1 -- start start 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa8e073740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x55897176fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa8e073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x558971769ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa8e073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa8e073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa841ab520 con 0x55897176d9b0 2024-04-25T01:27:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa8e073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa841a8df0 con 0x558971768840 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa8e073740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa841a6c40 con 0x558971767e50 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45234/0 (socket says 172.21.15.16:45234) 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 -- 172.21.15.16:0/3807388831 learned_addr learned my addr 172.21.15.16:0/3807388831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa888fe640 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x558971769ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa890ff640 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x55897176fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 -- 172.21.15.16:0/3807388831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 msgr2=0x55897176fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x55897176fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 -- 172.21.15.16:0/3807388831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 msgr2=0x558971769ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x558971769ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 -- 172.21.15.16:0/3807388831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558971763060 con 0x55897176d9b0 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa89900640 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176a1e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7faa7c056850 tx=0x7faa7c02af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a75d6a4f314be9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa6effd640 1 -- 172.21.15.16:0/3807388831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c051030 con 0x55897176d9b0 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa6effd640 1 -- 172.21.15.16:0/3807388831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa7c036040 con 0x55897176d9b0 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.466+0000 7faa6effd640 1 -- 172.21.15.16:0/3807388831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c056e30 con 0x55897176d9b0 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 -- 172.21.15.16:0/3807388831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 msgr2=0x55897176a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176a1e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7faa7c056850 tx=0x7faa7c02af40 comp rx=0 tx=0).stop 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 -- 172.21.15.16:0/3807388831 shutdown_connections 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55897176d9b0 0x55897176a1e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971768840 0x558971769ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 --2- 172.21.15.16:0/3807388831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558971767e50 0x55897176fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 -- 172.21.15.16:0/3807388831 >> 172.21.15.16:0/3807388831 conn(0x5589717634c0 msgr2=0x5589717638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 -- 172.21.15.16:0/3807388831 shutdown_connections 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 -- 172.21.15.16:0/3807388831 wait complete. 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 -- start start 2024-04-25T01:27:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.467+0000 7faa8e073740 1 Processor -- start 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971776040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 0x558971776580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55897176d9b0 0x558971777560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa841a72b0 con 0x558971768840 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa841a8720 con 0x558971767e50 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa841a7980 con 0x55897176d9b0 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa8e073740 10 client.0 _ll_register_callbacks cb 0x7faa7c005f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 0x558971776580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 0x558971776580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45246/0 (socket says 172.21.15.16:45246) 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 -- 172.21.15.16:0/2748437106 learned_addr learned my addr 172.21.15.16:0/2748437106 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa890ff640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971776040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa89900640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55897176d9b0 0x558971777560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 -- 172.21.15.16:0/2748437106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55897176d9b0 msgr2=0x558971777560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55897176d9b0 0x558971777560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 -- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 msgr2=0x558971776040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971776040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.468+0000 7faa888fe640 1 -- 172.21.15.16:0/2748437106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897176d700 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa888fe640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 0x558971776580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x558971764430 tx=0x7faa74005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebefd34fec78c946 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa6cff9640 1 -- 172.21.15.16:0/2748437106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74004020 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa6cff9640 1 -- 172.21.15.16:0/2748437106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa740041c0 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa7c036040 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa6cff9640 1 -- 172.21.15.16:0/2748437106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faa740043c0 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 10 client.15318 fetch_fsmap learned FSMap version 22 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558971778d30 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa6cff9640 1 -- 172.21.15.16:0/2748437106 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa7c036040 con 0x558971768840 2024-04-25T01:27:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa6cff9640 1 -- 172.21.15.16:0/2748437106 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faa74003790 con 0x558971768840 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 10 client.15318 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 10 client.15318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 20 client.15318 populate_metadata read hostname 'smithi016' 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5589717e49c0 0x5589717e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 4 client.15318 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.469+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5589716518e0 con 0x5589717e49c0 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.470+0000 7faa890ff640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5589717e49c0 0x5589717e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.470+0000 7faa890ff640 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5589717e49c0 0x5589717e6de0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f01acd7b1ee8dd6 server_cookie=c4b593aec7125634 in_seq=0 out_seq=0 2024-04-25T01:27:48.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.470+0000 7faa6cff9640 10 client.15318 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa6cff9640 1 -- 172.21.15.16:0/2748437106 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5589716518e0 con 0x5589717e49c0 2024-04-25T01:27:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa6cff9640 10 client.15318 handle_command_reply: tid=0 2024-04-25T01:27:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 client.15318 shutdown 2024-04-25T01:27:48.455 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 msgr2=0x558971776580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 0x558971776580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x558971764430 tx=0x7faa74005270 comp rx=0 tx=0).stop 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 shutdown_connections 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55897176d9b0 0x558971777560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558971768840 0x558971776580 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5589717e49c0 0x5589717e6de0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 --2- 172.21.15.16:0/2748437106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558971767e50 0x558971776040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa6cff9640 0 client.15318 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 >> 172.21.15.16:0/2748437106 conn(0x5589717634c0 msgr2=0x5589717721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 shutdown_connections 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 1 -- 172.21.15.16:0/2748437106 wait complete. 2024-04-25T01:27:48.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.474+0000 7faa8e073740 20 client.15318 trim_cache size 0 max 16384 2024-04-25T01:27:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:27:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:27:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:27:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa700a2cd0 msgr2=0x7faa700a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa700a2cd0 0x7faa700a5190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009a60 tx=0x7faa7c0095c0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 msgr2=0x7faa841978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 0x7faa841978d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7faa8000d950 tx=0x7faa8000de10 comp rx=0 tx=0).stop 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 shutdown_connections 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa700a2cd0 0x7faa700a5190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa84102250 0x7faa8419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa84101890 0x7faa84197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 --2- 172.21.15.16:0/1221775985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa841006a0 0x7faa841978d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 >> 172.21.15.16:0/1221775985 conn(0x7faa840fbc10 msgr2=0x7faa840fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 shutdown_connections 2024-04-25T01:27:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:27:48.475+0000 7faa8b363640 1 -- 172.21.15.16:0/1221775985 wait complete. 2024-04-25T01:27:49.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:49 smithi156 ceph-mon[32631]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-25T01:27:49.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:49 smithi016 ceph-mon[29153]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-25T01:27:49.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:49 smithi038 ceph-mon[32841]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-25T01:27:51.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:51 smithi156 ceph-mon[32631]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T01:27:51.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:51 smithi016 ceph-mon[29153]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T01:27:51.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:51 smithi038 ceph-mon[32841]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T01:27:53.500 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:53 smithi156 ceph-mon[32631]: pgmap v373: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T01:27:53.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:53 smithi016 ceph-mon[29153]: pgmap v373: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T01:27:53.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:53 smithi038 ceph-mon[32841]: pgmap v373: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T01:27:55.499 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:55 smithi156 ceph-mon[32631]: pgmap v374: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:27:55.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:55 smithi016 ceph-mon[29153]: pgmap v374: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:27:55.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:55 smithi038 ceph-mon[32841]: pgmap v374: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:27:57.499 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:57 smithi156 ceph-mon[32631]: pgmap v375: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:27:57.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:57 smithi016 ceph-mon[29153]: pgmap v375: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:27:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:57 smithi038 ceph-mon[32841]: pgmap v375: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:27:59.499 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:27:59 smithi156 ceph-mon[32631]: pgmap v376: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:27:59.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:27:59 smithi016 ceph-mon[29153]: pgmap v376: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:27:59.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:27:59 smithi038 ceph-mon[32841]: pgmap v376: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 -- 172.21.15.16:0/980151035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01017d0 msgr2=0x7f07d0101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 --2- 172.21.15.16:0/980151035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01017d0 0x7f07d0101bf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009900 tx=0x7f07cc02a410 comp rx=0 tx=0).stop 2024-04-25T01:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 -- 172.21.15.16:0/980151035 shutdown_connections 2024-04-25T01:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 --2- 172.21.15.16:0/980151035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07d0103380 0x7f07d0109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 --2- 172.21.15.16:0/980151035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07d01029c0 0x7f07d0102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 --2- 172.21.15.16:0/980151035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01017d0 0x7f07d0101bf0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 -- 172.21.15.16:0/980151035 >> 172.21.15.16:0/980151035 conn(0x7f07d00fcd60 msgr2=0x7f07d00ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 -- 172.21.15.16:0/980151035 shutdown_connections 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.571+0000 7f07d76dd640 1 -- 172.21.15.16:0/980151035 wait complete. 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 Processor -- start 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 -- start start 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07d01017d0 0x7f07d00754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 0x7f07d0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07d0103380 0x7f07d0074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d010d320 con 0x7f07d01029c0 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07d010d1a0 con 0x7f07d0103380 2024-04-25T01:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d76dd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f07d010d4a0 con 0x7f07d01017d0 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d4c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 0x7f07d0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d4c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 0x7f07d0073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58122/0 (socket says 172.21.15.16:58122) 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d4c78640 1 -- 172.21.15.16:0/621292248 learned_addr learned my addr 172.21.15.16:0/621292248 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d5479640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07d01017d0 0x7f07d00754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.572+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07d0103380 0x7f07d0074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d4c78640 1 -- 172.21.15.16:0/621292248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07d01017d0 msgr2=0x7f07d00754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d4c78640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07d01017d0 0x7f07d00754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d4c78640 1 -- 172.21.15.16:0/621292248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07d0103380 msgr2=0x7f07d0074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d4c78640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07d0103380 0x7f07d0074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d4c78640 1 -- 172.21.15.16:0/621292248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d0074900 con 0x7f07d01029c0 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d4c78640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 0x7f07d0073b30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f07c000d950 tx=0x7f07c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07c67fc640 1 -- 172.21.15.16:0/621292248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0017030 con 0x7f07d01029c0 2024-04-25T01:28:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d01a1a50 con 0x7f07d01029c0 2024-04-25T01:28:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07c67fc640 1 -- 172.21.15.16:0/621292248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07c0004020 con 0x7f07d01029c0 2024-04-25T01:28:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07c67fc640 1 -- 172.21.15.16:0/621292248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c00047a0 con 0x7f07d01029c0 2024-04-25T01:28:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.573+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d01a1fa0 con 0x7f07d01029c0 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.575+0000 7f07c67fc640 1 -- 172.21.15.16:0/621292248 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f07c00041d0 con 0x7f07d01029c0 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.576+0000 7f07c67fc640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07bc0a2cd0 0x7f07bc0a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.576+0000 7f07d5479640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07bc0a2cd0 0x7f07bc0a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.576+0000 7f07c67fc640 1 -- 172.21.15.16:0/621292248 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f07c00ced80 con 0x7f07d01029c0 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.576+0000 7f07d5479640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07bc0a2cd0 0x7f07bc0a5190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f07cc005ab0 tx=0x7f07cc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 Processor -- start 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 -- start start 2024-04-25T01:28:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e50b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e510340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 0x55de6e512e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55de6e46ecb0 con 0x55de6e5109b0 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55de6e4175b0 con 0x55de6e50ae50 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07da3ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55de6e46ac70 con 0x55de6e50b840 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 0x55de6e512e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 0x55de6e512e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58132/0 (socket says 172.21.15.16:58132) 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 -- 172.21.15.16:0/4127343245 learned_addr learned my addr 172.21.15.16:0/4127343245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d4c78640 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e510340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5479640 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e50b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 -- 172.21.15.16:0/4127343245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 msgr2=0x55de6e510340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e510340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 -- 172.21.15.16:0/4127343245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 msgr2=0x55de6e50b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e50b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5c7a640 1 -- 172.21.15.16:0/4127343245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de6e183210 con 0x55de6e5109b0 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.577+0000 7f07d5479640 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e50b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.578+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 0x55de6e512e40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009f40 tx=0x7f07c8010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5e6076aaa58692 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.578+0000 7f07bb7fe640 1 -- 172.21.15.16:0/4127343245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8003b70 con 0x55de6e5109b0 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.578+0000 7f07bb7fe640 1 -- 172.21.15.16:0/4127343245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07c8003d10 con 0x55de6e5109b0 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.578+0000 7f07bb7fe640 1 -- 172.21.15.16:0/4127343245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80118e0 con 0x55de6e5109b0 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 -- 172.21.15.16:0/4127343245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 msgr2=0x55de6e512e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 0x55de6e512e40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009f40 tx=0x7f07c8010860 comp rx=0 tx=0).stop 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 -- 172.21.15.16:0/4127343245 shutdown_connections 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e5109b0 0x55de6e512e40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e510340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 --2- 172.21.15.16:0/4127343245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e50b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 -- 172.21.15.16:0/4127343245 >> 172.21.15.16:0/4127343245 conn(0x55de6e5064c0 msgr2=0x55de6e5068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 -- 172.21.15.16:0/4127343245 shutdown_connections 2024-04-25T01:28:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 -- 172.21.15.16:0/4127343245 wait complete. 2024-04-25T01:28:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 -- start start 2024-04-25T01:28:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.579+0000 7f07da3ea740 1 Processor -- start 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e5152a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 0x55de6e5157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55de6e513960 con 0x55de6e50b840 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55de6e42ab50 con 0x55de6e50ae50 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55de6e513ae0 con 0x55de6e5109b0 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07da3ea740 10 client.0 _ll_register_callbacks cb 0x55de6e465ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d5479640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e5152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 0x55de6e5157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 0x55de6e5157e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58146/0 (socket says 172.21.15.16:58146) 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d5479640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e5152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54100/0 (socket says 172.21.15.16:54100) 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 -- 172.21.15.16:0/1325350613 learned_addr learned my addr 172.21.15.16:0/1325350613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 -- 172.21.15.16:0/1325350613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 msgr2=0x55de6e51c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 -- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 msgr2=0x55de6e5152a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e5152a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d4c78640 1 -- 172.21.15.16:0/1325350613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de6e24f620 con 0x55de6e50b840 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.580+0000 7f07d5479640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e5152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07d4c78640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 0x55de6e5157e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f07c009b100 tx=0x7f07c009b2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb20875c5caed716 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07b8ff9640 1 -- 172.21.15.16:0/1325350613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c00b4060 con 0x55de6e50b840 2024-04-25T01:28:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07b8ff9640 1 -- 172.21.15.16:0/1325350613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f07c00b4200 con 0x55de6e50b840 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07c8003d10 con 0x55de6e50b840 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07b8ff9640 1 -- 172.21.15.16:0/1325350613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f07c00b4400 con 0x55de6e50b840 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 10 client.15330 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55de6e51fb70 con 0x55de6e50b840 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07b8ff9640 1 -- 172.21.15.16:0/1325350613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07c00a4070 con 0x55de6e50b840 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07b8ff9640 1 -- 172.21.15.16:0/1325350613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f07c0009a30 con 0x55de6e50b840 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 10 client.15330 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 10 client.15330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 20 client.15330 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e5211f0 0x55de6e523620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 4 client.15330 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.581+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f07c8003d10 con 0x55de6e5211f0 2024-04-25T01:28:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.582+0000 7f07d5479640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e5211f0 0x55de6e523620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.582+0000 7f07d5479640 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e5211f0 0x55de6e523620 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a06165123309198 server_cookie=b719c1249b4a4b0e in_seq=0 out_seq=0 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.582+0000 7f07b8ff9640 10 client.15330 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07b8ff9640 1 -- 172.21.15.16:0/1325350613 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f07c8003d10 con 0x55de6e5211f0 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07b8ff9640 10 client.15330 handle_command_reply: tid=0 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 client.15330 shutdown 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 msgr2=0x55de6e5157e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 0x55de6e5157e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f07c009b100 tx=0x7f07c009b2f0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 shutdown_connections 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50b840 0x55de6e5157e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e5211f0 0x55de6e523620 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 --2- 172.21.15.16:0/1325350613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50ae50 0x55de6e5152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07b8ff9640 0 client.15330 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 >> 172.21.15.16:0/1325350613 conn(0x55de6e5064c0 msgr2=0x55de6e5055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 shutdown_connections 2024-04-25T01:28:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.583+0000 7f07da3ea740 1 -- 172.21.15.16:0/1325350613 wait complete. 2024-04-25T01:28:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.584+0000 7f07da3ea740 20 client.15330 trim_cache size 0 max 16384 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 Processor -- start 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 -- start start 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e512da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e5068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e5109b0 0x55de6e50cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01ad780 con 0x55de6e50ae50 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01ab050 con 0x55de6e5109b0 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07da3ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01a8ea0 con 0x55de6e50b840 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07d5479640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e512da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07d5479640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e512da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58156/0 (socket says 172.21.15.16:58156) 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.612+0000 7f07d5479640 1 -- 172.21.15.16:0/2064084722 learned_addr learned my addr 172.21.15.16:0/2064084722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e5109b0 0x55de6e50cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d4c78640 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e5068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5479640 1 -- 172.21.15.16:0/2064084722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 msgr2=0x55de6e5068a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5479640 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e5068a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5479640 1 -- 172.21.15.16:0/2064084722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e5109b0 msgr2=0x55de6e50cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5479640 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e5109b0 0x55de6e50cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5479640 1 -- 172.21.15.16:0/2064084722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de6e50bc20 con 0x55de6e50ae50 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07d5479640 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e512da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f07cc000c00 tx=0x7f07cc03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396bd7404852f858 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07baffd640 1 -- 172.21.15.16:0/2064084722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc035050 con 0x55de6e50ae50 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07baffd640 1 -- 172.21.15.16:0/2064084722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07cc0527d0 con 0x55de6e50ae50 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.613+0000 7f07baffd640 1 -- 172.21.15.16:0/2064084722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc031e20 con 0x55de6e50ae50 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 -- 172.21.15.16:0/2064084722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 msgr2=0x55de6e512da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e512da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f07cc000c00 tx=0x7f07cc03e000 comp rx=0 tx=0).stop 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 -- 172.21.15.16:0/2064084722 shutdown_connections 2024-04-25T01:28:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e5109b0 0x55de6e50cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e50b840 0x55de6e5068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2064084722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e512da0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 -- 172.21.15.16:0/2064084722 >> 172.21.15.16:0/2064084722 conn(0x55de6e5064c0 msgr2=0x55de6e5012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 -- 172.21.15.16:0/2064084722 shutdown_connections 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 -- 172.21.15.16:0/2064084722 wait complete. 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 -- start start 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.614+0000 7f07da3ea740 1 Processor -- start 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e518e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50b840 0x55de6e519340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01a9510 con 0x55de6e50ae50 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01aa980 con 0x55de6e50b840 2024-04-25T01:28:01.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01a9be0 con 0x55de6e5109b0 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07da3ea740 10 client.0 _ll_register_callbacks cb 0x7f07c00976a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e518e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e518e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58164/0 (socket says 172.21.15.16:58164) 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 -- 172.21.15.16:0/2274337299 learned_addr learned my addr 172.21.15.16:0/2274337299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5c7a640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d4c78640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50b840 0x55de6e519340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 -- 172.21.15.16:0/2274337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 msgr2=0x55de6e51a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 -- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50b840 msgr2=0x55de6e519340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50b840 0x55de6e519340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 -- 172.21.15.16:0/2274337299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de6e51a940 con 0x55de6e50ae50 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07d5479640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e518e00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f07cc02aa00 tx=0x7f07cc02ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39239486394a804a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.615+0000 7f07b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07b8ff9640 1 -- 172.21.15.16:0/2274337299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc035070 con 0x55de6e50ae50 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07b8ff9640 1 -- 172.21.15.16:0/2274337299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f07cc052a40 con 0x55de6e50ae50 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07cc0527d0 con 0x55de6e50ae50 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07b8ff9640 1 -- 172.21.15.16:0/2274337299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f07cc052320 con 0x55de6e50ae50 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 10 client.15336 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55de6e51bc50 con 0x55de6e50ae50 2024-04-25T01:28:01.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07b8ff9640 1 -- 172.21.15.16:0/2274337299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07cc0527d0 con 0x55de6e50ae50 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07b8ff9640 1 -- 172.21.15.16:0/2274337299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f07cc003be0 con 0x55de6e50ae50 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 10 client.15336 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 10 client.15336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 20 client.15336 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e587b90 0x55de6e589fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 4 client.15336 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.616+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de6e47c1f0 con 0x55de6e587b90 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.617+0000 7f07d4c78640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e587b90 0x55de6e589fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.617+0000 7f07d4c78640 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e587b90 0x55de6e589fb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85799572cca11f0d server_cookie=50e9804460237ae5 in_seq=0 out_seq=0 2024-04-25T01:28:01.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.617+0000 7f07b8ff9640 10 client.15336 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07b8ff9640 1 -- 172.21.15.16:0/2274337299 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de6e47c1f0 con 0x55de6e587b90 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07b8ff9640 10 client.15336 handle_command_reply: tid=0 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 client.15336 shutdown 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 msgr2=0x55de6e518e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e518e00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f07cc02aa00 tx=0x7f07cc02ab90 comp rx=0 tx=0).stop 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 shutdown_connections 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de6e5109b0 0x55de6e51a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55de6e587b90 0x55de6e589fb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de6e50b840 0x55de6e519340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07b8ff9640 0 client.15336 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 --2- 172.21.15.16:0/2274337299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55de6e50ae50 0x55de6e518e00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 >> 172.21.15.16:0/2274337299 conn(0x55de6e5064c0 msgr2=0x55de6e5068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 shutdown_connections 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 1 -- 172.21.15.16:0/2274337299 wait complete. 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.621+0000 7f07da3ea740 20 client.15336 trim_cache size 0 max 16384 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:28:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:28:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07bc0a2cd0 msgr2=0x7f07bc0a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07bc0a2cd0 0x7f07bc0a5190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f07cc005ab0 tx=0x7f07cc005a20 comp rx=0 tx=0).stop 2024-04-25T01:28:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 msgr2=0x7f07d0073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 0x7f07d0073b30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f07c000d950 tx=0x7f07c000de10 comp rx=0 tx=0).stop 2024-04-25T01:28:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 shutdown_connections 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07bc0a2cd0 0x7f07bc0a5190 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07d0103380 0x7f07d0074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07d01029c0 0x7f07d0073b30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 --2- 172.21.15.16:0/621292248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f07d01017d0 0x7f07d00754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 >> 172.21.15.16:0/621292248 conn(0x7f07d00fcd60 msgr2=0x7f07d0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 shutdown_connections 2024-04-25T01:28:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:01.622+0000 7f07d76dd640 1 -- 172.21.15.16:0/621292248 wait complete. 2024-04-25T01:28:01.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:01 smithi016 ceph-mon[29153]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:01.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:01 smithi038 ceph-mon[32841]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:01.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:01 smithi156 ceph-mon[32631]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:03.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:03 smithi016 ceph-mon[29153]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:03.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:03 smithi038 ceph-mon[32841]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:03.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:03 smithi156 ceph-mon[32631]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:05.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:05 smithi016 ceph-mon[29153]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:28:05.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:05 smithi038 ceph-mon[32841]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:28:05.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:05 smithi156 ceph-mon[32631]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:28:07.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:07 smithi016 ceph-mon[29153]: pgmap v380: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:07.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:07 smithi038 ceph-mon[32841]: pgmap v380: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:07.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:07 smithi156 ceph-mon[32631]: pgmap v380: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:09.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:09 smithi016 ceph-mon[29153]: pgmap v381: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:09.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:09 smithi038 ceph-mon[32841]: pgmap v381: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:09.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:09 smithi156 ceph-mon[32631]: pgmap v381: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:11.552 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:11 smithi156 ceph-mon[32631]: pgmap v382: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:11.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:11 smithi016 ceph-mon[29153]: pgmap v382: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:11.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:11 smithi038 ceph-mon[32841]: pgmap v382: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:13.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:13 smithi016 ceph-mon[29153]: pgmap v383: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:13.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:13 smithi038 ceph-mon[32841]: pgmap v383: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:13.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:13 smithi156 ceph-mon[32631]: pgmap v383: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 -- 172.21.15.16:0/1158274309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 msgr2=0x7f21b8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 --2- 172.21.15.16:0/1158274309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b8100ac0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009900 tx=0x7f21b402a410 comp rx=0 tx=0).stop 2024-04-25T01:28:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 -- 172.21.15.16:0/1158274309 shutdown_connections 2024-04-25T01:28:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 --2- 172.21.15.16:0/1158274309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 0x7f21b8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 --2- 172.21.15.16:0/1158274309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 0x7f21b8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 --2- 172.21.15.16:0/1158274309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b8100ac0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 -- 172.21.15.16:0/1158274309 >> 172.21.15.16:0/1158274309 conn(0x7f21b80fbc10 msgr2=0x7f21b80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 -- 172.21.15.16:0/1158274309 shutdown_connections 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 -- 172.21.15.16:0/1158274309 wait complete. 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 Processor -- start 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.752+0000 7f21c053b640 1 -- start start 2024-04-25T01:28:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21c053b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21c053b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 0x7f21b8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b81979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48532/0 (socket says 172.21.15.16:48532) 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21c053b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 0x7f21b819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21c053b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21b810c270 con 0x7f21b81006a0 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21c053b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f21b810c0f0 con 0x7f21b8101890 2024-04-25T01:28:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21c053b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21b810c3f0 con 0x7f21b8102250 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 -- 172.21.15.16:0/1951080251 learned_addr learned my addr 172.21.15.16:0/1951080251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21bdad6640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 0x7f21b8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 -- 172.21.15.16:0/1951080251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 msgr2=0x7f21b819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21bead8640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 0x7f21b819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 0x7f21b819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 -- 172.21.15.16:0/1951080251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 msgr2=0x7f21b8197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 0x7f21b8197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21be2d7640 1 -- 172.21.15.16:0/1951080251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b819f5b0 con 0x7f21b81006a0 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21bdad6640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 0x7f21b8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.753+0000 7f21bead8640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 0x7f21b819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.754+0000 7f21be2d7640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b81979e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f21b40314e0 tx=0x7f21b4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.754+0000 7f21af7fe640 1 -- 172.21.15.16:0/1951080251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b403f070 con 0x7f21b81006a0 2024-04-25T01:28:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.754+0000 7f21af7fe640 1 -- 172.21.15.16:0/1951080251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21b4031e40 con 0x7f21b81006a0 2024-04-25T01:28:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.754+0000 7f21af7fe640 1 -- 172.21.15.16:0/1951080251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4035030 con 0x7f21b81006a0 2024-04-25T01:28:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.754+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b819f840 con 0x7f21b81006a0 2024-04-25T01:28:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.754+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b819fd60 con 0x7f21b81006a0 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.757+0000 7f21af7fe640 1 -- 172.21.15.16:0/1951080251 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f21b4042070 con 0x7f21b81006a0 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 Processor -- start 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 -- start start 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76c6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21af7fe640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21a40a2d50 0x7f21a40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6c50 0x5640c76cb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5640c7625cb0 con 0x5640c76c6c50 2024-04-25T01:28:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5640c75ce5b0 con 0x5640c76c6260 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21c3246740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5640c7621c70 con 0x5640c76cbdc0 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21be2d7640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21a40a2d50 0x7f21a40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21bead8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21bead8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:34796/0 (socket says 172.21.15.16:34796) 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.758+0000 7f21bead8640 1 -- 172.21.15.16:0/2127197998 learned_addr learned my addr 172.21.15.16:0/2127197998 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6c50 0x5640c76cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 -- 172.21.15.16:0/2127197998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 msgr2=0x5640c76ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 -- 172.21.15.16:0/2127197998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6c50 msgr2=0x5640c76cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6c50 0x5640c76cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 -- 172.21.15.16:0/2127197998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640c733a210 con 0x5640c76c6260 2024-04-25T01:28:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6c50 0x5640c76cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21af7fe640 1 -- 172.21.15.16:0/1951080251 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f21b40efc20 con 0x7f21b81006a0 2024-04-25T01:28:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21bead8640 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76c6680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21b0014bc0 tx=0x7f21b000a6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58c5534122a5f717 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21be2d7640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21a40a2d50 0x7f21a40a5210 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f21b40d2800 tx=0x7f21b40b7e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21acbfe640 1 -- 172.21.15.16:0/2127197998 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0014250 con 0x5640c76c6260 2024-04-25T01:28:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21acbfe640 1 -- 172.21.15.16:0/2127197998 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21b0011400 con 0x5640c76c6260 2024-04-25T01:28:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.759+0000 7f21acbfe640 1 -- 172.21.15.16:0/2127197998 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0013650 con 0x5640c76c6260 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 -- 172.21.15.16:0/2127197998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 msgr2=0x5640c76c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76c6680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21b0014bc0 tx=0x7f21b000a6b0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 -- 172.21.15.16:0/2127197998 shutdown_connections 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6c50 0x5640c76cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 --2- 172.21.15.16:0/2127197998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76c6680 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 -- 172.21.15.16:0/2127197998 >> 172.21.15.16:0/2127197998 conn(0x5640c76bd4c0 msgr2=0x5640c76bd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 -- 172.21.15.16:0/2127197998 shutdown_connections 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.760+0000 7f21c3246740 1 -- 172.21.15.16:0/2127197998 wait complete. 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 -- start start 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 Processor -- start 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6260 0x5640c76d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21bead8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21bead8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48538/0 (socket says 172.21.15.16:48538) 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21bead8640 1 -- 172.21.15.16:0/2973413768 learned_addr learned my addr 172.21.15.16:0/2973413768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5640c76ced70 con 0x5640c76cbdc0 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5640c75e1b50 con 0x5640c76c6c50 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5640c76ceef0 con 0x5640c76c6260 2024-04-25T01:28:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.761+0000 7f21bead8640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6260 0x5640c76d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21c3246740 10 client.0 _ll_register_callbacks cb 0x5640c761cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bead8640 1 -- 172.21.15.16:0/2973413768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6260 msgr2=0x5640c76d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bead8640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6260 0x5640c76d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bead8640 1 -- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 msgr2=0x5640c76d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bead8640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bead8640 1 -- 172.21.15.16:0/2973413768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640c7406620 con 0x5640c76cbdc0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21bead8640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f21b000afd0 tx=0x7f21b00145b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84dea780584ef1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21a1ffb640 1 -- 172.21.15.16:0/2973413768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00117b0 con 0x5640c76cbdc0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21a1ffb640 1 -- 172.21.15.16:0/2973413768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f21b0021430 con 0x5640c76cbdc0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b0011400 con 0x5640c76cbdc0 2024-04-25T01:28:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21a1ffb640 1 -- 172.21.15.16:0/2973413768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f21b0021630 con 0x5640c76cbdc0 2024-04-25T01:28:14.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21c3246740 10 client.15348 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:14.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.762+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5640c76dada0 con 0x5640c76cbdc0 2024-04-25T01:28:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21a1ffb640 1 -- 172.21.15.16:0/2973413768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21b0011400 con 0x5640c76cbdc0 2024-04-25T01:28:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21a1ffb640 1 -- 172.21.15.16:0/2973413768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f21b0010040 con 0x5640c76cbdc0 2024-04-25T01:28:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21c3246740 10 client.15348 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21c3246740 10 client.15348 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21c3246740 20 client.15348 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21c3246740 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c76dc1a0 0x5640c76de590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21c3246740 4 client.15348 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640c76331f0 con 0x5640c76dc1a0 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.763+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c76dc1a0 0x5640c76de590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.764+0000 7f21a1ffb640 10 client.15348 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.764+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c76dc1a0 0x5640c76de590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0b96b09f761d15b server_cookie=17a5d50f31f6dee4 in_seq=0 out_seq=0 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.764+0000 7f21a1ffb640 1 -- 172.21.15.16:0/2973413768 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640c76331f0 con 0x5640c76dc1a0 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.764+0000 7f21a1ffb640 10 client.15348 handle_command_reply: tid=0 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.764+0000 7f21c3246740 1 client.15348 shutdown 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 msgr2=0x5640c76d7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f21b000afd0 tx=0x7f21b00145b0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 shutdown_connections 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c76dc1a0 0x5640c76de590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21a1ffb640 0 client.15348 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 --2- 172.21.15.16:0/2973413768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6260 0x5640c76d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 >> 172.21.15.16:0/2973413768 conn(0x5640c76bd4c0 msgr2=0x5640c76bc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 shutdown_connections 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 1 -- 172.21.15.16:0/2973413768 wait complete. 2024-04-25T01:28:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.765+0000 7f21c3246740 20 client.15348 trim_cache size 0 max 16384 2024-04-25T01:28:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 Processor -- start 2024-04-25T01:28:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 -- start start 2024-04-25T01:28:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76e3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6c50 0x5640c76e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81a9380 con 0x5640c76cbdc0 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81a6c50 con 0x5640c76c6260 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21c3246740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81a4aa0 con 0x5640c76c6c50 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21bead8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21bead8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48540/0 (socket says 172.21.15.16:48540) 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.780+0000 7f21bead8640 1 -- 172.21.15.16:0/2456603623 learned_addr learned my addr 172.21.15.16:0/2456603623 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76e3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6c50 0x5640c76e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 -- 172.21.15.16:0/2456603623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6c50 msgr2=0x5640c76e42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6c50 0x5640c76e42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 -- 172.21.15.16:0/2456603623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 msgr2=0x5640c76e3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76e3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 -- 172.21.15.16:0/2456603623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640c76b7850 con 0x5640c76cbdc0 2024-04-25T01:28:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bdad6640 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6c50 0x5640c76e42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21bead8640 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x5640c76cbd20 tx=0x7f21b000af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365f9c9624c4c06e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21a3fff640 1 -- 172.21.15.16:0/2456603623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0011d00 con 0x5640c76cbdc0 2024-04-25T01:28:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21a3fff640 1 -- 172.21.15.16:0/2456603623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21b0035040 con 0x5640c76cbdc0 2024-04-25T01:28:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.781+0000 7f21a3fff640 1 -- 172.21.15.16:0/2456603623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b001eb20 con 0x5640c76cbdc0 2024-04-25T01:28:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 -- 172.21.15.16:0/2456603623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 msgr2=0x5640c76d7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x5640c76cbd20 tx=0x7f21b000af40 comp rx=0 tx=0).stop 2024-04-25T01:28:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 -- 172.21.15.16:0/2456603623 shutdown_connections 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76cbdc0 0x5640c76d7a00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76c6c50 0x5640c76e42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 --2- 172.21.15.16:0/2456603623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6260 0x5640c76e3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 -- 172.21.15.16:0/2456603623 >> 172.21.15.16:0/2456603623 conn(0x5640c76bd4c0 msgr2=0x5640c76cb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 -- 172.21.15.16:0/2456603623 shutdown_connections 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.782+0000 7f21c3246740 1 -- 172.21.15.16:0/2456603623 wait complete. 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 -- start start 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 Processor -- start 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 0x5640c76d4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76d6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81a5110 con 0x5640c76c6260 2024-04-25T01:28:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81a6580 con 0x5640c76c6c50 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21b81a57e0 con 0x5640c76cbdc0 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21c3246740 10 client.0 _ll_register_callbacks cb 0x7f21b0014500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 0x5640c76d4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 0x5640c76d4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48542/0 (socket says 172.21.15.16:48542) 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 -- 172.21.15.16:0/3234857099 learned_addr learned my addr 172.21.15.16:0/3234857099 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bdad6640 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 -- 172.21.15.16:0/3234857099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 msgr2=0x5640c76d6030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76d6030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 -- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 msgr2=0x5640c76d51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.783+0000 7f21bead8640 1 -- 172.21.15.16:0/3234857099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640c76d6620 con 0x5640c76c6260 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21bead8640 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 0x5640c76d4c60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000f80 tx=0x7f21b0014b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b355410a7a953 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21a1ffb640 1 -- 172.21.15.16:0/3234857099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0034040 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21a1ffb640 1 -- 172.21.15.16:0/3234857099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f21b002a070 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b0035040 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21a1ffb640 1 -- 172.21.15.16:0/3234857099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f21b0014630 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21c3246740 10 client.15354 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5640c76d78a0 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21a1ffb640 1 -- 172.21.15.16:0/3234857099 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21b0035040 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21a1ffb640 1 -- 172.21.15.16:0/3234857099 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f21b0021960 con 0x5640c76c6260 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21c3246740 10 client.15354 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21c3246740 10 client.15354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.784+0000 7f21c3246740 20 client.15354 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.785+0000 7f21c3246740 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c77428a0 0x5640c7744c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.785+0000 7f21c3246740 4 client.15354 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:28:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.785+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640c762d540 con 0x5640c77428a0 2024-04-25T01:28:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.785+0000 7f21bdad6640 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c77428a0 0x5640c7744c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.785+0000 7f21bdad6640 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c77428a0 0x5640c7744c90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6da3cee204392c09 server_cookie=50c50be22e0b6ec6 in_seq=0 out_seq=0 2024-04-25T01:28:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.785+0000 7f21a1ffb640 10 client.15354 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21a1ffb640 1 -- 172.21.15.16:0/3234857099 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640c762d540 con 0x5640c77428a0 2024-04-25T01:28:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21a1ffb640 10 client.15354 handle_command_reply: tid=0 2024-04-25T01:28:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 client.15354 shutdown 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 msgr2=0x5640c76d4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 0x5640c76d4c60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000f80 tx=0x7f21b0014b70 comp rx=0 tx=0).stop 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 shutdown_connections 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640c76cbdc0 0x5640c76d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5640c77428a0 0x5640c7744c90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640c76c6c50 0x5640c76d51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21a1ffb640 0 client.15354 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 --2- 172.21.15.16:0/3234857099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5640c76c6260 0x5640c76d4c60 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.789+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 >> 172.21.15.16:0/3234857099 conn(0x5640c76bd4c0 msgr2=0x5640c76cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 shutdown_connections 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c3246740 1 -- 172.21.15.16:0/3234857099 wait complete. 2024-04-25T01:28:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c3246740 20 client.15354 trim_cache size 0 max 16384 2024-04-25T01:28:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:28:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:28:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:28:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21a40a2d50 msgr2=0x7f21a40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21a40a2d50 0x7f21a40a5210 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f21b40d2800 tx=0x7f21b40b7e10 comp rx=0 tx=0).stop 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 msgr2=0x7f21b81979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b81979e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f21b40314e0 tx=0x7f21b4003d50 comp rx=0 tx=0).stop 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 shutdown_connections 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f21a40a2d50 0x7f21a40a5210 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21b8102250 0x7f21b819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.790+0000 7f21c053b640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21b8101890 0x7f21b8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.791+0000 7f21c053b640 1 --2- 172.21.15.16:0/1951080251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21b81006a0 0x7f21b81979e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.791+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 >> 172.21.15.16:0/1951080251 conn(0x7f21b80fbc10 msgr2=0x7f21b80fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.791+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 shutdown_connections 2024-04-25T01:28:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:14.791+0000 7f21c053b640 1 -- 172.21.15.16:0/1951080251 wait complete. 2024-04-25T01:28:15.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:15 smithi016 ceph-mon[29153]: pgmap v384: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:15.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:15 smithi038 ceph-mon[32841]: pgmap v384: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:15.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:15 smithi156 ceph-mon[32631]: pgmap v384: 97 pgs: 97 active+clean; 663 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:17.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:17 smithi016 ceph-mon[29153]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:28:17.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:17 smithi038 ceph-mon[32841]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:28:17.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:17 smithi156 ceph-mon[32631]: pgmap v385: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:28:19.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:19 smithi016 ceph-mon[29153]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:19.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:19 smithi038 ceph-mon[32841]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:19.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:19 smithi156 ceph-mon[32631]: pgmap v386: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:21.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:21 smithi016 ceph-mon[29153]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:21.718 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:21 smithi038 ceph-mon[32841]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:21 smithi156 ceph-mon[32631]: pgmap v387: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T01:28:23.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:23 smithi016 ceph-mon[29153]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:23.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:23 smithi038 ceph-mon[32841]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:23.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:23 smithi156 ceph-mon[32631]: pgmap v388: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:25.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:25 smithi016 ceph-mon[29153]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:25.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:25 smithi038 ceph-mon[32841]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:25.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:25 smithi156 ceph-mon[32631]: pgmap v389: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T01:28:27.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:27 smithi016 ceph-mon[29153]: pgmap v390: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:27.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:27 smithi038 ceph-mon[32841]: pgmap v390: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:27.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:27 smithi156 ceph-mon[32631]: pgmap v390: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T01:28:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 -- 172.21.15.16:0/70368366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb01006a0 msgr2=0x7fbeb0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 --2- 172.21.15.16:0/70368366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb01006a0 0x7fbeb0100ac0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009900 tx=0x7fbea002a410 comp rx=0 tx=0).stop 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 -- 172.21.15.16:0/70368366 shutdown_connections 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 --2- 172.21.15.16:0/70368366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0102250 0x7fbeb0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 --2- 172.21.15.16:0/70368366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb0101890 0x7fbeb0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 --2- 172.21.15.16:0/70368366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb01006a0 0x7fbeb0100ac0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 -- 172.21.15.16:0/70368366 >> 172.21.15.16:0/70368366 conn(0x7fbeb00fbc10 msgr2=0x7fbeb00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 -- 172.21.15.16:0/70368366 shutdown_connections 2024-04-25T01:28:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.880+0000 7fbeb4970640 1 -- 172.21.15.16:0/70368366 wait complete. 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.881+0000 7fbeb4970640 1 Processor -- start 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.881+0000 7fbeb4970640 1 -- start start 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.881+0000 7fbeb4970640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 0x7fbeb0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.881+0000 7fbeb4970640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0101890 0x7fbeb0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.881+0000 7fbeb4970640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb0102250 0x7fbeb019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeb4970640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb010c1f0 con 0x7fbeb0102250 2024-04-25T01:28:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeb4970640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb010c070 con 0x7fbeb01006a0 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeb4970640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb010c370 con 0x7fbeb0101890 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 0x7fbeb0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeaed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb0102250 0x7fbeb019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 0x7fbeb0197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55830/0 (socket says 172.21.15.16:55830) 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 -- 172.21.15.16:0/2904594856 learned_addr learned my addr 172.21.15.16:0/2904594856 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0101890 0x7fbeb0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 -- 172.21.15.16:0/2904594856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0101890 msgr2=0x7fbeb0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0101890 0x7fbeb0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 -- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb0102250 msgr2=0x7fbeb019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb0102250 0x7fbeb019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 -- 172.21.15.16:0/2904594856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb019f5e0 con 0x7fbeb01006a0 2024-04-25T01:28:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0101890 0x7fbeb0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb0102250 0x7fbeb019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.882+0000 7fbeae59c640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 0x7fbeb0197a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbea00314e0 tx=0x7fbea0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.883+0000 7fbe9f7fe640 1 -- 172.21.15.16:0/2904594856 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0035070 con 0x7fbeb01006a0 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.883+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb0102ab0 con 0x7fbeb01006a0 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.883+0000 7fbe9f7fe640 1 -- 172.21.15.16:0/2904594856 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbea0003c20 con 0x7fbeb01006a0 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.883+0000 7fbe9f7fe640 1 -- 172.21.15.16:0/2904594856 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0047620 con 0x7fbeb01006a0 2024-04-25T01:28:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.883+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb0102ff0 con 0x7fbeb01006a0 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.885+0000 7fbe9f7fe640 1 -- 172.21.15.16:0/2904594856 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fbea0031580 con 0x7fbeb01006a0 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbe9f7fe640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe980a2cd0 0x7fbe980a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 Processor -- start 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe980a2cd0 0x7fbe980a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbe9f7fe640 1 -- 172.21.15.16:0/2904594856 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbea00f79f0 con 0x7fbeb01006a0 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 -- start start 2024-04-25T01:28:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 0x55e2b7092270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b7097340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e2b6ff5cb0 con 0x55e2b7092840 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.886+0000 7fbeb7322740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e2b6f9e5b0 con 0x55e2b7091e50 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeb7322740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e2b6ff1c70 con 0x55e2b70979b0 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 0x55e2b7092270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 0x55e2b7092270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55842/0 (socket says 172.21.15.16:55842) 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 -- 172.21.15.16:0/2849030033 learned_addr learned my addr 172.21.15.16:0/2849030033 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe980a2cd0 0x7fbe980a5190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbea40058c0 tx=0x7fbea4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b7097340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 -- 172.21.15.16:0/2849030033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 msgr2=0x55e2b7099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 -- 172.21.15.16:0/2849030033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 msgr2=0x55e2b7097340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b7097340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 -- 172.21.15.16:0/2849030033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2b6d0a210 con 0x55e2b7091e50 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.887+0000 7fbeae59c640 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 0x55e2b7092270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbea00c0a40 tx=0x7fbea00c3c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13ec90afd0a14f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbe9cff9640 1 -- 172.21.15.16:0/2849030033 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00c8420 con 0x55e2b7091e50 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbe9cff9640 1 -- 172.21.15.16:0/2849030033 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbea00ccce0 con 0x55e2b7091e50 2024-04-25T01:28:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbe9cff9640 1 -- 172.21.15.16:0/2849030033 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00c06e0 con 0x55e2b7091e50 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 -- 172.21.15.16:0/2849030033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 msgr2=0x55e2b7092270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 0x55e2b7092270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbea00c0a40 tx=0x7fbea00c3c70 comp rx=0 tx=0).stop 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 -- 172.21.15.16:0/2849030033 shutdown_connections 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b7097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 --2- 172.21.15.16:0/2849030033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7091e50 0x55e2b7092270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 -- 172.21.15.16:0/2849030033 >> 172.21.15.16:0/2849030033 conn(0x55e2b708d4c0 msgr2=0x55e2b708d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 -- 172.21.15.16:0/2849030033 shutdown_connections 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.888+0000 7fbeb7322740 1 -- 172.21.15.16:0/2849030033 wait complete. 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 -- start start 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 Processor -- start 2024-04-25T01:28:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b709c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b709c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b70979b0 0x55e2b70a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e2b709a720 con 0x55e2b7092840 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e2b6fb1b50 con 0x55e2b70979b0 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e2b709a8a0 con 0x55e2b7091e50 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeadd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b709c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeadd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b709c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57994/0 (socket says 172.21.15.16:57994) 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeadd9b640 1 -- 172.21.15.16:0/3279143085 learned_addr learned my addr 172.21.15.16:0/3279143085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b70979b0 0x55e2b70a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.889+0000 7fbeb7322740 10 client.0 _ll_register_callbacks cb 0x55e2b6fecca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeae59c640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b709c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeadd9b640 1 -- 172.21.15.16:0/3279143085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 msgr2=0x55e2b709c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b709c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeadd9b640 1 -- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b70979b0 msgr2=0x55e2b70a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b70979b0 0x55e2b70a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeadd9b640 1 -- 172.21.15.16:0/3279143085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2b6dd6620 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeae59c640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b709c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbea00ccce0 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b709c560 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbea40116f0 tx=0x7fbea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9627a04fb2cca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbe95ffb640 10 client.15366 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbe95ffb640 1 -- 172.21.15.16:0/3279143085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4004220 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbe95ffb640 1 -- 172.21.15.16:0/3279143085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbea40043c0 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbe95ffb640 1 -- 172.21.15.16:0/3279143085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fbea401e850 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeb7322740 10 client.15366 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.890+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55e2b70a6970 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbe95ffb640 1 -- 172.21.15.16:0/3279143085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fbea401e9f0 con 0x55e2b7092840 2024-04-25T01:28:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeb7322740 10 client.15366 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeb7322740 10 client.15366 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeb7322740 20 client.15366 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeb7322740 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b70aa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeb7322740 4 client.15366 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2b70031f0 con 0x55e2b70a8060 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.891+0000 7fbeae59c640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b70aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbeae59c640 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b70aa450 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=326a0f3ed8765c39 server_cookie=d807f3f184248664 in_seq=0 out_seq=0 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbe95ffb640 10 client.15366 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbe95ffb640 1 -- 172.21.15.16:0/3279143085 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2b70031f0 con 0x55e2b70a8060 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbe95ffb640 10 client.15366 handle_command_reply: tid=0 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbeb7322740 1 client.15366 shutdown 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 msgr2=0x55e2b709c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.875 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbeb7322740 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b709c560 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbea40116f0 tx=0x7fbea40046c0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.892+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 shutdown_connections 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b70aa450 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b70979b0 0x55e2b70a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbe95ffb640 0 client.15366 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7092840 0x55e2b709c560 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 --2- 172.21.15.16:0/3279143085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b709c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 >> 172.21.15.16:0/3279143085 conn(0x55e2b708d4c0 msgr2=0x55e2b708c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 shutdown_connections 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 1 -- 172.21.15.16:0/3279143085 wait complete. 2024-04-25T01:28:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.893+0000 7fbeb7322740 20 client.15366 trim_cache size 0 max 16384 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.922+0000 7fbeb7322740 1 Processor -- start 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.922+0000 7fbeb7322740 1 -- start start 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 0x55e2b708cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7090470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 0x55e2b708cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeb7322740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb01a9330 con 0x55e2b7091e50 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeae59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 0x55e2b708cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58000/0 (socket says 172.21.15.16:58000) 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeae59c640 1 -- 172.21.15.16:0/424721513 learned_addr learned my addr 172.21.15.16:0/424721513 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeb7322740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb01a6c00 con 0x55e2b7092840 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeb7322740 1 -- 172.21.15.16:0/424721513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb01a4a50 con 0x55e2b70979b0 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7090470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeadd9b640 1 -- 172.21.15.16:0/424721513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 msgr2=0x55e2b7090470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7090470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeadd9b640 1 -- 172.21.15.16:0/424721513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 msgr2=0x55e2b708cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 0x55e2b708cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeadd9b640 1 -- 172.21.15.16:0/424721513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2b70aa7c0 con 0x55e2b7092840 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.923+0000 7fbeae59c640 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 0x55e2b708cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.924+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a3560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55e2b709d500 tx=0x7fbea4021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd11cb5b624825b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.924+0000 7fbe97fff640 1 -- 172.21.15.16:0/424721513 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea400bdb0 con 0x55e2b7092840 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.924+0000 7fbe97fff640 1 -- 172.21.15.16:0/424721513 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbea00ccce0 con 0x55e2b7092840 2024-04-25T01:28:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.924+0000 7fbe97fff640 1 -- 172.21.15.16:0/424721513 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea401f690 con 0x55e2b7092840 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 -- 172.21.15.16:0/424721513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 msgr2=0x55e2b70a3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a3560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55e2b709d500 tx=0x7fbea4021000 comp rx=0 tx=0).stop 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 -- 172.21.15.16:0/424721513 shutdown_connections 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b70979b0 0x55e2b7090470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a3560 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 --2- 172.21.15.16:0/424721513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b7091e50 0x55e2b708cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 -- 172.21.15.16:0/424721513 >> 172.21.15.16:0/424721513 conn(0x55e2b708d4c0 msgr2=0x55e2b7093d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 -- 172.21.15.16:0/424721513 shutdown_connections 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.925+0000 7fbeb7322740 1 -- 172.21.15.16:0/424721513 wait complete. 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 -- start start 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 Processor -- start 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b70a0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 0x55e2b70a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb01a50c0 con 0x55e2b70979b0 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb01a6530 con 0x55e2b7092840 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeb7322740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb01a5790 con 0x55e2b7091e50 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeaed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 0x55e2b70a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.926+0000 7fbeaed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 0x55e2b70a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58002/0 (socket says 172.21.15.16:58002) 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 -- 172.21.15.16:0/252273009 learned_addr learned my addr 172.21.15.16:0/252273009 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeae59c640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b70a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeadd9b640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeb7322740 10 client.0 _ll_register_callbacks cb 0x7fbea400b930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 -- 172.21.15.16:0/252273009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 msgr2=0x55e2b70a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b70a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 -- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 msgr2=0x55e2b70a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 -- 172.21.15.16:0/252273009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2b70a23a0 con 0x55e2b70979b0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbea00ccce0 con 0x55e2b70979b0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeaed9d640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 0x55e2b70a1db0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbea8011ae0 tx=0x7fbea8010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76e1276394777a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbe95ffb640 10 client.15372 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbe95ffb640 1 -- 172.21.15.16:0/252273009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea8021060 con 0x55e2b70979b0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbe95ffb640 1 -- 172.21.15.16:0/252273009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbea8003970 con 0x55e2b70979b0 2024-04-25T01:28:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbe95ffb640 1 -- 172.21.15.16:0/252273009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fbea8003bb0 con 0x55e2b70979b0 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeb7322740 10 client.15372 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.927+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55e2b70a3680 con 0x55e2b70979b0 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbe95ffb640 1 -- 172.21.15.16:0/252273009 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fbea801b030 con 0x55e2b70979b0 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeb7322740 10 client.15372 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeb7322740 10 client.15372 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeb7322740 20 client.15372 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeb7322740 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b71109a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeb7322740 4 client.15372 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e2b6ffd540 con 0x55e2b70a8060 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.928+0000 7fbeae59c640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b71109a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.929+0000 7fbeae59c640 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b71109a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68dc7f4f24e7432 server_cookie=af18dffed7d2fd82 in_seq=0 out_seq=0 2024-04-25T01:28:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.929+0000 7fbe95ffb640 10 client.15372 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbe95ffb640 1 -- 172.21.15.16:0/252273009 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e2b6ffd540 con 0x55e2b70a8060 2024-04-25T01:28:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbe95ffb640 10 client.15372 handle_command_reply: tid=0 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 client.15372 shutdown 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 msgr2=0x55e2b70a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 0x55e2b70a1db0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbea8011ae0 tx=0x7fbea8010430 comp rx=0 tx=0).stop 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 shutdown_connections 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e2b70979b0 0x55e2b70a1db0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e2b70a8060 0x55e2b71109a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2b7092840 0x55e2b70a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 --2- 172.21.15.16:0/252273009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2b7091e50 0x55e2b70a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 >> 172.21.15.16:0/252273009 conn(0x55e2b708d4c0 msgr2=0x55e2b7090470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbe95ffb640 0 client.15372 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.932+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 shutdown_connections 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb7322740 1 -- 172.21.15.16:0/252273009 wait complete. 2024-04-25T01:28:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb7322740 20 client.15372 trim_cache size 0 max 16384 2024-04-25T01:28:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:28:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:28:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:28:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe980a2cd0 msgr2=0x7fbe980a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe980a2cd0 0x7fbe980a5190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbea40058c0 tx=0x7fbea4005850 comp rx=0 tx=0).stop 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 msgr2=0x7fbeb0197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 0x7fbeb0197a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbea00314e0 tx=0x7fbea0003820 comp rx=0 tx=0).stop 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 shutdown_connections 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbe980a2cd0 0x7fbe980a5190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbeb0102250 0x7fbeb019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbeb0101890 0x7fbeb0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 --2- 172.21.15.16:0/2904594856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbeb01006a0 0x7fbeb0197a10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 >> 172.21.15.16:0/2904594856 conn(0x7fbeb00fbc10 msgr2=0x7fbeb00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 shutdown_connections 2024-04-25T01:28:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:27.933+0000 7fbeb4970640 1 -- 172.21.15.16:0/2904594856 wait complete. 2024-04-25T01:28:29.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:29 smithi016 ceph-mon[29153]: pgmap v391: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T01:28:29.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:29 smithi038 ceph-mon[32841]: pgmap v391: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T01:28:29.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:29 smithi156 ceph-mon[32631]: pgmap v391: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T01:28:31.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:31 smithi016 ceph-mon[29153]: pgmap v392: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-25T01:28:31.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:31 smithi016 ceph-mon[29153]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T01:28:31.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:31 smithi038 ceph-mon[32841]: pgmap v392: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-25T01:28:31.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:31 smithi038 ceph-mon[32841]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T01:28:31.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:31 smithi156 ceph-mon[32631]: pgmap v392: 97 pgs: 97 active+clean; 664 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-25T01:28:31.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:31 smithi156 ceph-mon[32631]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T01:28:32.982 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 96% (12044/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-04-25T01:28:33.174 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:33.175 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T01:28:33.178 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T01:28:33.242 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T01:28:33.244 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:28:33.244 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T01:28:33.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:33 smithi016 ceph-mon[29153]: pgmap v393: 97 pgs: 97 active+clean; 665 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:28:33.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:33 smithi038 ceph-mon[32841]: pgmap v393: 97 pgs: 97 active+clean; 665 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:28:33.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:33 smithi156 ceph-mon[32631]: pgmap v393: 97 pgs: 97 active+clean; 665 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:28:33.848 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T01:28:33.902 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T01:28:33.943 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:28:33.945 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:28:33.945 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T01:28:33.985 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:28:33.988 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:28:33.988 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T01:28:34.045 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-04-25T01:28:34.046 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-04-25T01:28:34.046 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-04-25T01:28:34.110 INFO:tasks.workunit.client.0.smithi016.stdout:getting iogen 2024-04-25T01:28:34.111 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'getting iogen' 2024-04-25T01:28:34.111 INFO:tasks.workunit.client.0.smithi016.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-25T01:28:34.142 INFO:tasks.workunit.client.0.smithi016.stderr:--2024-04-25 01:28:34-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-25T01:28:34.143 INFO:tasks.workunit.client.0.smithi016.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-25T01:28:34.164 INFO:tasks.workunit.client.0.smithi016.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-25T01:28:34.185 INFO:tasks.workunit.client.0.smithi016.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-25T01:28:34.185 INFO:tasks.workunit.client.0.smithi016.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-04-25T01:28:34.186 INFO:tasks.workunit.client.0.smithi016.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-04-25T01:28:34.189 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:34.190 INFO:tasks.workunit.client.0.smithi016.stderr: 0K ....... 100% 1.23G=0s 2024-04-25T01:28:34.190 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:34.190 INFO:tasks.workunit.client.0.smithi016.stderr:2024-04-25 01:28:34 (1.23 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-04-25T01:28:34.190 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-25T01:28:34.190 INFO:tasks.workunit.client.0.smithi016.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-04-25T01:28:34.201 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0 2024-04-25T01:28:34.202 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/src 2024-04-25T01:28:34.204 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/src/iogen.8 2024-04-25T01:28:34.206 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/src/iogen.c 2024-04-25T01:28:34.209 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/README 2024-04-25T01:28:34.212 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/Makefile.bsd 2024-04-25T01:28:34.214 INFO:tasks.workunit.client.0.smithi016.stdout:iogen_3.1p0/Makefile 2024-04-25T01:28:34.217 INFO:tasks.workunit.client.0.smithi016.stderr:+ cd iogen_3.1p0 2024-04-25T01:28:34.218 INFO:tasks.workunit.client.0.smithi016.stdout:making iogen 2024-04-25T01:28:34.218 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'making iogen' 2024-04-25T01:28:34.218 INFO:tasks.workunit.client.0.smithi016.stderr:+ make 2024-04-25T01:28:34.230 INFO:tasks.workunit.client.0.smithi016.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-04-25T01:28:34.468 INFO:tasks.workunit.client.0.smithi016.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-04-25T01:28:34.567 INFO:tasks.workunit.client.0.smithi016.stdout:running iogen 2024-04-25T01:28:34.568 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'running iogen' 2024-04-25T01:28:34.568 INFO:tasks.workunit.client.0.smithi016.stderr:+ ./iogen -n 5 -s 2g 2024-04-25T01:28:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 81979 2024-04-25T01:28:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 81980 2024-04-25T01:28:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 81981 2024-04-25T01:28:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 81982 2024-04-25T01:28:34.569 INFO:tasks.workunit.client.0.smithi016.stderr:child spawned 81983 2024-04-25T01:28:34.570 INFO:tasks.workunit.client.0.smithi016.stdout:sleep for 10 min 2024-04-25T01:28:34.570 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'sleep for 10 min' 2024-04-25T01:28:34.570 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 600 2024-04-25T01:28:35.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:35 smithi016 ceph-mon[29153]: pgmap v394: 97 pgs: 97 active+clean; 665 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T01:28:35.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:35 smithi038 ceph-mon[32841]: pgmap v394: 97 pgs: 97 active+clean; 665 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T01:28:35.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:35 smithi156 ceph-mon[32631]: pgmap v394: 97 pgs: 97 active+clean; 665 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T01:28:37.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:37 smithi016 ceph-mon[29153]: pgmap v395: 97 pgs: 97 active+clean; 787 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-25T01:28:37.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:37 smithi038 ceph-mon[32841]: pgmap v395: 97 pgs: 97 active+clean; 787 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-25T01:28:37.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:37 smithi156 ceph-mon[32631]: pgmap v395: 97 pgs: 97 active+clean; 787 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-04-25T01:28:39.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:39 smithi016 ceph-mon[29153]: pgmap v396: 97 pgs: 97 active+clean; 181 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-25T01:28:39.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:39 smithi038 ceph-mon[32841]: pgmap v396: 97 pgs: 97 active+clean; 181 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-25T01:28:39.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:39 smithi156 ceph-mon[32631]: pgmap v396: 97 pgs: 97 active+clean; 181 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-25T01:28:40.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:40 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:40.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:40 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:40.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:40 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 -- 172.21.15.16:0/884208236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540102990 msgr2=0x7f2540102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 --2- 172.21.15.16:0/884208236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540102990 0x7f2540102e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2534009860 tx=0x7f253402a340 comp rx=0 tx=0).stop 2024-04-25T01:28:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 -- 172.21.15.16:0/884208236 shutdown_connections 2024-04-25T01:28:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 --2- 172.21.15.16:0/884208236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2540103350 0x7f2540109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 --2- 172.21.15.16:0/884208236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540102990 0x7f2540102e10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 --2- 172.21.15.16:0/884208236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25401017a0 0x7f2540101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 -- 172.21.15.16:0/884208236 >> 172.21.15.16:0/884208236 conn(0x7f25400fcd30 msgr2=0x7f25400ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 -- 172.21.15.16:0/884208236 shutdown_connections 2024-04-25T01:28:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.069+0000 7f2544b09640 1 -- 172.21.15.16:0/884208236 wait complete. 2024-04-25T01:28:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.070+0000 7f2544b09640 1 Processor -- start 2024-04-25T01:28:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.070+0000 7f2544b09640 1 -- start start 2024-04-25T01:28:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.070+0000 7f2544b09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25401017a0 0x7f25400754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f2544b09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2540102990 0x7f2540073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f2544b09640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540103350 0x7f2540074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f2544b09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f254010d2d0 con 0x7f2540102990 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f2544b09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f254010d150 con 0x7f25401017a0 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f2544b09640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f254010d450 con 0x7f2540103350 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2540102990 0x7f2540073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2540102990 0x7f2540073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53970/0 (socket says 172.21.15.16:53970) 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f253dd9b640 1 -- 172.21.15.16:0/269021468 learned_addr learned my addr 172.21.15.16:0/269021468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f253e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25401017a0 0x7f25400754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.071+0000 7f253ed9d640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540103350 0x7f2540074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f253e59c640 1 -- 172.21.15.16:0/269021468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540103350 msgr2=0x7f2540074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f253e59c640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540103350 0x7f2540074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f253e59c640 1 -- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2540102990 msgr2=0x7f2540073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f253e59c640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2540102990 0x7f2540073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f253e59c640 1 -- 172.21.15.16:0/269021468 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2540074900 con 0x7f25401017a0 2024-04-25T01:28:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f253e59c640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25401017a0 0x7f25400754f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25401027f0 tx=0x7f253000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f252f7fe640 1 -- 172.21.15.16:0/269021468 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530004440 con 0x7f25401017a0 2024-04-25T01:28:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f252f7fe640 1 -- 172.21.15.16:0/269021468 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253000cd70 con 0x7f25401017a0 2024-04-25T01:28:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.072+0000 7f252f7fe640 1 -- 172.21.15.16:0/269021468 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530019480 con 0x7f25401017a0 2024-04-25T01:28:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.073+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2540197570 con 0x7f25401017a0 2024-04-25T01:28:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.073+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2540197a40 con 0x7f25401017a0 2024-04-25T01:28:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.076+0000 7f252f7fe640 1 -- 172.21.15.16:0/269021468 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2530003710 con 0x7f25401017a0 2024-04-25T01:28:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f252f7fe640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25280a2cb0 0x7f25280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 Processor -- start 2024-04-25T01:28:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f252f7fe640 1 -- 172.21.15.16:0/269021468 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f25300cf3e0 con 0x7f25401017a0 2024-04-25T01:28:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 -- start start 2024-04-25T01:28:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f253dd9b640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25280a2cb0 0x7f25280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 0x562dfc3d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3d6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d69b0 0x562dfc3d8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562dfc334cb0 con 0x562dfc3d1840 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.077+0000 7f254780d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562dfc2dd5b0 con 0x562dfc3d0e50 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f254780d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562dfc330c70 con 0x562dfc3d69b0 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 0x562dfc3d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 0x562dfc3d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:42342/0 (socket says 172.21.15.16:42342) 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 -- 172.21.15.16:0/3648237936 learned_addr learned my addr 172.21.15.16:0/3648237936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253dd9b640 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3d6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253ed9d640 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d69b0 0x562dfc3d8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 -- 172.21.15.16:0/3648237936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d69b0 msgr2=0x562dfc3d8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d69b0 0x562dfc3d8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 -- 172.21.15.16:0/3648237936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 msgr2=0x562dfc3d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3d6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253e59c640 1 -- 172.21.15.16:0/3648237936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dfc049210 con 0x562dfc3d0e50 2024-04-25T01:28:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.078+0000 7f253dd9b640 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3d6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:28:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.079+0000 7f253e59c640 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 0x562dfc3d1270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25300985f0 tx=0x7f2530009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88070ebaee970f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.079+0000 7f252cff9640 1 -- 172.21.15.16:0/3648237936 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253009bc20 con 0x562dfc3d0e50 2024-04-25T01:28:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.079+0000 7f252cff9640 1 -- 172.21.15.16:0/3648237936 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253009ffd0 con 0x562dfc3d0e50 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f252cff9640 1 -- 172.21.15.16:0/3648237936 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300a4660 con 0x562dfc3d0e50 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 -- 172.21.15.16:0/3648237936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 msgr2=0x562dfc3d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 0x562dfc3d1270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25300985f0 tx=0x7f2530009510 comp rx=0 tx=0).stop 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f253dd9b640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25280a2cb0 0x7f25280a5170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f253402a930 tx=0x7f2534009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 -- 172.21.15.16:0/3648237936 shutdown_connections 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d69b0 0x562dfc3d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3d6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 --2- 172.21.15.16:0/3648237936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d0e50 0x562dfc3d1270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.080+0000 7f254780d740 1 -- 172.21.15.16:0/3648237936 >> 172.21.15.16:0/3648237936 conn(0x562dfc3cc4c0 msgr2=0x562dfc3cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.081+0000 7f254780d740 1 -- 172.21.15.16:0/3648237936 shutdown_connections 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.081+0000 7f254780d740 1 -- 172.21.15.16:0/3648237936 wait complete. 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.081+0000 7f254780d740 1 -- start start 2024-04-25T01:28:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.081+0000 7f254780d740 1 Processor -- start 2024-04-25T01:28:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3db270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3e27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562dfc3d9940 con 0x562dfc3d1840 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562dfc2f0b50 con 0x562dfc3d69b0 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562dfc3d9ac0 con 0x562dfc3d0e50 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f254780d740 10 client.0 _ll_register_callbacks cb 0x562dfc32bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f253e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3db270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f253ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3e27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f253ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3e27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:42354/0 (socket says 172.21.15.16:42354) 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f253ed9d640 1 -- 172.21.15.16:0/2213154642 learned_addr learned my addr 172.21.15.16:0/2213154642 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.082+0000 7f253dd9b640 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f253ed9d640 1 -- 172.21.15.16:0/2213154642 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 msgr2=0x562dfc3db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f253ed9d640 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3db270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f253ed9d640 1 -- 172.21.15.16:0/2213154642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 msgr2=0x562dfc3db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f253ed9d640 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3db7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f253ed9d640 1 -- 172.21.15.16:0/2213154642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dfc115620 con 0x562dfc3d69b0 2024-04-25T01:28:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f253ed9d640 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3e27f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f253800e620 tx=0x7f253800eae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67919557ef82e5eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f2525ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f2525ffb640 1 -- 172.21.15.16:0/2213154642 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538003e40 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253009ffd0 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f2525ffb640 1 -- 172.21.15.16:0/2213154642 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2538014230 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.083+0000 7f2525ffb640 1 -- 172.21.15.16:0/2213154642 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2538026050 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 10 client.25061 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562dfc3f6ba0 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f2525ffb640 1 -- 172.21.15.16:0/2213154642 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253009ffd0 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f2525ffb640 1 -- 172.21.15.16:0/2213154642 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f253800c580 con 0x562dfc3d69b0 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 10 client.25061 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 10 client.25061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 20 client.25061 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc3f8700 0x562dfc3fab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 4 client.25061 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dfc3421f0 con 0x562dfc3f8700 2024-04-25T01:28:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.084+0000 7f253e59c640 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc3f8700 0x562dfc3fab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.085+0000 7f2525ffb640 10 client.25061 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.085+0000 7f253e59c640 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc3f8700 0x562dfc3fab20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fdecfd7fac46140 server_cookie=c31076cc4f90138b in_seq=0 out_seq=0 2024-04-25T01:28:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.085+0000 7f2525ffb640 1 -- 172.21.15.16:0/2213154642 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562dfc3421f0 con 0x562dfc3f8700 2024-04-25T01:28:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.085+0000 7f2525ffb640 10 client.25061 handle_command_reply: tid=0 2024-04-25T01:28:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 client.25061 shutdown 2024-04-25T01:28:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 msgr2=0x562dfc3e27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3e27f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f253800e620 tx=0x7f253800eae0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 shutdown_connections 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc3f8700 0x562dfc3fab20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3e27f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3db7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f2525ffb640 0 client.25061 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 --2- 172.21.15.16:0/2213154642 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3db270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 >> 172.21.15.16:0/2213154642 conn(0x562dfc3cc4c0 msgr2=0x562dfc3cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 shutdown_connections 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 1 -- 172.21.15.16:0/2213154642 wait complete. 2024-04-25T01:28:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.086+0000 7f254780d740 20 client.25061 trim_cache size 0 max 16384 2024-04-25T01:28:41.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.117+0000 7f254780d740 1 Processor -- start 2024-04-25T01:28:41.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 -- start start 2024-04-25T01:28:41.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3cad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3cb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25401a91e0 con 0x562dfc3d1840 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25401a6ab0 con 0x562dfc3d69b0 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f254780d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25401a4900 con 0x562dfc3d0e50 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3e27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53988/0 (socket says 172.21.15.16:53988) 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 -- 172.21.15.16:0/128811622 learned_addr learned my addr 172.21.15.16:0/128811622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253ed9d640 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253e59c640 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 -- 172.21.15.16:0/128811622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 msgr2=0x562dfc3cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 -- 172.21.15.16:0/128811622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 msgr2=0x562dfc3cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.118+0000 7f253dd9b640 1 -- 172.21.15.16:0/128811622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dfc3cc060 con 0x562dfc3d1840 2024-04-25T01:28:41.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.119+0000 7f253dd9b640 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3e27b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2534035040 tx=0x7f253402a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadaed6f2acbcfa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:41.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.119+0000 7f2527fff640 1 -- 172.21.15.16:0/128811622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534037040 con 0x562dfc3d1840 2024-04-25T01:28:41.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.119+0000 7f2527fff640 1 -- 172.21.15.16:0/128811622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2534050070 con 0x562dfc3d1840 2024-04-25T01:28:41.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.119+0000 7f2527fff640 1 -- 172.21.15.16:0/128811622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534032820 con 0x562dfc3d1840 2024-04-25T01:28:41.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 -- 172.21.15.16:0/128811622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 msgr2=0x562dfc3e27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3e27b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2534035040 tx=0x7f253402a340 comp rx=0 tx=0).stop 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 -- 172.21.15.16:0/128811622 shutdown_connections 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d69b0 0x562dfc3cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d1840 0x562dfc3e27b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 --2- 172.21.15.16:0/128811622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 -- 172.21.15.16:0/128811622 >> 172.21.15.16:0/128811622 conn(0x562dfc3cc4c0 msgr2=0x562dfc3db270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 -- 172.21.15.16:0/128811622 shutdown_connections 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.120+0000 7f254780d740 1 -- 172.21.15.16:0/128811622 wait complete. 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 -- start start 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 Processor -- start 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d1840 0x562dfc3dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d69b0 0x562dfc3e0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25401a4f70 con 0x562dfc3d69b0 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25401a63e0 con 0x562dfc3d1840 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25401a5640 con 0x562dfc3d0e50 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f253ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d69b0 0x562dfc3e0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f253e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f254780d740 10 client.0 _ll_register_callbacks cb 0x7f25380099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d1840 0x562dfc3dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f253e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3dfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:53884/0 (socket says 172.21.15.16:53884) 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.121+0000 7f253e59c640 1 -- 172.21.15.16:0/1213973194 learned_addr learned my addr 172.21.15.16:0/1213973194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253ed9d640 1 -- 172.21.15.16:0/1213973194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 msgr2=0x562dfc3dfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253ed9d640 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3dfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253ed9d640 1 -- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d1840 msgr2=0x562dfc3dffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253ed9d640 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d1840 0x562dfc3dffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253ed9d640 1 -- 172.21.15.16:0/1213973194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dfc3e15e0 con 0x562dfc3d69b0 2024-04-25T01:28:41.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253dd9b640 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d1840 0x562dfc3dffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:41.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f253ed9d640 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d69b0 0x562dfc3e0ff0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f253800e5f0 tx=0x7f253800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c7c9217bafa599 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:41.104 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f2525ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:41.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f2525ffb640 1 -- 172.21.15.16:0/1213973194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538014430 con 0x562dfc3d69b0 2024-04-25T01:28:41.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f2525ffb640 1 -- 172.21.15.16:0/1213973194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f25380145d0 con 0x562dfc3d69b0 2024-04-25T01:28:41.105 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2534050070 con 0x562dfc3d69b0 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.122+0000 7f2525ffb640 1 -- 172.21.15.16:0/1213973194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2538038070 con 0x562dfc3d69b0 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 10 client.15387 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562dfc3e2960 con 0x562dfc3d69b0 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f2525ffb640 1 -- 172.21.15.16:0/1213973194 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2534050070 con 0x562dfc3d69b0 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f2525ffb640 1 -- 172.21.15.16:0/1213973194 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f25380344c0 con 0x562dfc3d69b0 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 10 client.15387 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 10 client.15387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 20 client.15387 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc45f2e0 0x562dfc461700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 4 client.15387 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:28:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562dfc33c540 con 0x562dfc45f2e0 2024-04-25T01:28:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.123+0000 7f253e59c640 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc45f2e0 0x562dfc461700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.124+0000 7f2525ffb640 10 client.15387 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.124+0000 7f253e59c640 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc45f2e0 0x562dfc461700 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5d8d2d204923504 server_cookie=8b94653548b1bb09 in_seq=0 out_seq=0 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.141+0000 7f2525ffb640 1 -- 172.21.15.16:0/1213973194 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562dfc33c540 con 0x562dfc45f2e0 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.141+0000 7f2525ffb640 10 client.15387 handle_command_reply: tid=0 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.141+0000 7f254780d740 1 client.15387 shutdown 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d69b0 msgr2=0x562dfc3e0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d69b0 0x562dfc3e0ff0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f253800e5f0 tx=0x7f253800ce40 comp rx=0 tx=0).stop 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 shutdown_connections 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562dfc3d69b0 0x562dfc3e0ff0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562dfc45f2e0 0x562dfc461700 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f2525ffb640 0 client.15387 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562dfc3d1840 0x562dfc3dffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 --2- 172.21.15.16:0/1213973194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562dfc3d0e50 0x562dfc3dfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 >> 172.21.15.16:0/1213973194 conn(0x562dfc3cc4c0 msgr2=0x562dfc3cad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 shutdown_connections 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 1 -- 172.21.15.16:0/1213973194 wait complete. 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.142+0000 7f254780d740 20 client.15387 trim_cache size 0 max 16384 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:28:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25280a2cb0 msgr2=0x7f25280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25280a2cb0 0x7f25280a5170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f253402a930 tx=0x7f2534009490 comp rx=0 tx=0).stop 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25401017a0 msgr2=0x7f25400754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25401017a0 0x7f25400754f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25401027f0 tx=0x7f253000ee50 comp rx=0 tx=0).stop 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 shutdown_connections 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25280a2cb0 0x7f25280a5170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2540103350 0x7f2540074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2540102990 0x7f2540073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 --2- 172.21.15.16:0/269021468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25401017a0 0x7f25400754f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.143+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 >> 172.21.15.16:0/269021468 conn(0x7f25400fcd30 msgr2=0x7f2540105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.144+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 shutdown_connections 2024-04-25T01:28:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:41.144+0000 7f2544b09640 1 -- 172.21.15.16:0/269021468 wait complete. 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: pgmap v397: 97 pgs: 97 active+clean; 482 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 40 MiB/s wr, 30 op/s 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:41 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.650 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: pgmap v397: 97 pgs: 97 active+clean; 482 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 40 MiB/s wr, 30 op/s 2024-04-25T01:28:41.651 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.651 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.651 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.651 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.651 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.651 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:41 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: pgmap v397: 97 pgs: 97 active+clean; 482 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 40 MiB/s wr, 30 op/s 2024-04-25T01:28:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:41.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:41 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:42.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:42.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:42.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:42.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:42.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:42.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:42.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:42.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:42.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:28:43.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:43 smithi016 ceph-mon[29153]: pgmap v398: 97 pgs: 97 active+clean; 689 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 57 MiB/s wr, 42 op/s 2024-04-25T01:28:43.616 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:43 smithi038 ceph-mon[32841]: pgmap v398: 97 pgs: 97 active+clean; 689 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 57 MiB/s wr, 42 op/s 2024-04-25T01:28:43.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:43 smithi156 ceph-mon[32631]: pgmap v398: 97 pgs: 97 active+clean; 689 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 57 MiB/s wr, 42 op/s 2024-04-25T01:28:45.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:45 smithi016 ceph-mon[29153]: pgmap v399: 97 pgs: 97 active+clean; 1.2 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 103 MiB/s wr, 68 op/s 2024-04-25T01:28:45.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:45 smithi038 ceph-mon[32841]: pgmap v399: 97 pgs: 97 active+clean; 1.2 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 103 MiB/s wr, 68 op/s 2024-04-25T01:28:45.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:45 smithi156 ceph-mon[32631]: pgmap v399: 97 pgs: 97 active+clean; 1.2 GiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 103 MiB/s wr, 68 op/s 2024-04-25T01:28:47.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:47 smithi016 ceph-mon[29153]: pgmap v400: 97 pgs: 97 active+clean; 1.3 GiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 113 MiB/s wr, 77 op/s 2024-04-25T01:28:47.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:47 smithi038 ceph-mon[32841]: pgmap v400: 97 pgs: 97 active+clean; 1.3 GiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 113 MiB/s wr, 77 op/s 2024-04-25T01:28:47.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:47 smithi156 ceph-mon[32631]: pgmap v400: 97 pgs: 97 active+clean; 1.3 GiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 113 MiB/s wr, 77 op/s 2024-04-25T01:28:49.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:49 smithi016 ceph-mon[29153]: pgmap v401: 97 pgs: 97 active+clean; 1.7 GiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 146 MiB/s wr, 90 op/s 2024-04-25T01:28:49.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:49 smithi038 ceph-mon[32841]: pgmap v401: 97 pgs: 97 active+clean; 1.7 GiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 146 MiB/s wr, 90 op/s 2024-04-25T01:28:49.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:49 smithi156 ceph-mon[32631]: pgmap v401: 97 pgs: 97 active+clean; 1.7 GiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 146 MiB/s wr, 90 op/s 2024-04-25T01:28:51.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:51 smithi016 ceph-mon[29153]: pgmap v402: 97 pgs: 97 active+clean; 2.0 GiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 157 MiB/s wr, 96 op/s 2024-04-25T01:28:51.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:51 smithi038 ceph-mon[32841]: pgmap v402: 97 pgs: 97 active+clean; 2.0 GiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 157 MiB/s wr, 96 op/s 2024-04-25T01:28:51.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:51 smithi156 ceph-mon[32631]: pgmap v402: 97 pgs: 97 active+clean; 2.0 GiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 157 MiB/s wr, 96 op/s 2024-04-25T01:28:53.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:53 smithi016 ceph-mon[29153]: pgmap v403: 97 pgs: 97 active+clean; 2.3 GiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 157 MiB/s wr, 95 op/s 2024-04-25T01:28:53.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:53 smithi038 ceph-mon[32841]: pgmap v403: 97 pgs: 97 active+clean; 2.3 GiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 157 MiB/s wr, 95 op/s 2024-04-25T01:28:53.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:53 smithi156 ceph-mon[32631]: pgmap v403: 97 pgs: 97 active+clean; 2.3 GiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 157 MiB/s wr, 95 op/s 2024-04-25T01:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 -- 172.21.15.16:0/3264659517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d81006b0 msgr2=0x7f56d8100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3264659517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d81006b0 0x7f56d8100ad0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009900 tx=0x7f56d402a410 comp rx=0 tx=0).stop 2024-04-25T01:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 -- 172.21.15.16:0/3264659517 shutdown_connections 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3264659517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d8102260 0x7f56d8108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3264659517 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d81018a0 0x7f56d8101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3264659517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d81006b0 0x7f56d8100ad0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 -- 172.21.15.16:0/3264659517 >> 172.21.15.16:0/3264659517 conn(0x7f56d80fbc20 msgr2=0x7f56d80fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 -- 172.21.15.16:0/3264659517 shutdown_connections 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.283+0000 7f56dfc37640 1 -- 172.21.15.16:0/3264659517 wait complete. 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 Processor -- start 2024-04-25T01:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 -- start start 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d81006b0 0x7f56d81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d81018a0 0x7f56d8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 0x7f56d819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56d810c1e0 con 0x7f56d8102260 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f56d810c060 con 0x7f56d81018a0 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56dfc37640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56d810c360 con 0x7f56d81006b0 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56de1d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 0x7f56d819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.284+0000 7f56de1d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 0x7f56d819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34790/0 (socket says 172.21.15.16:34790) 2024-04-25T01:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 -- 172.21.15.16:0/3578211618 learned_addr learned my addr 172.21.15.16:0/3578211618 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d81018a0 0x7f56d8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d81006b0 0x7f56d81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 -- 172.21.15.16:0/3578211618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d81006b0 msgr2=0x7f56d81979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d81006b0 0x7f56d81979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 -- 172.21.15.16:0/3578211618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d81018a0 msgr2=0x7f56d8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d81018a0 0x7f56d8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 -- 172.21.15.16:0/3578211618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d819f5c0 con 0x7f56d8102260 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d81018a0 0x7f56d8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56de1d4640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 0x7f56d819ef50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f56d000e9b0 tx=0x7f56d000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56caffd640 1 -- 172.21.15.16:0/3578211618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d000cd80 con 0x7f56d8102260 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.285+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56d819f8b0 con 0x7f56d8102260 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.286+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56d819fda0 con 0x7f56d8102260 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.286+0000 7f56caffd640 1 -- 172.21.15.16:0/3578211618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56d00035f0 con 0x7f56d8102260 2024-04-25T01:28:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.286+0000 7f56caffd640 1 -- 172.21.15.16:0/3578211618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0010690 con 0x7f56d8102260 2024-04-25T01:28:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 Processor -- start 2024-04-25T01:28:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56caffd640 1 -- 172.21.15.16:0/3578211618 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f56d0010830 con 0x7f56d8102260 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 -- start start 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbd2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbd7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 0x5607efbda250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5607efb31cb0 con 0x5607efbd2260 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5607efada5b0 con 0x5607efbd7dc0 2024-04-25T01:28:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.289+0000 7f56e2a17740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5607efb2dc70 con 0x5607efbd2c50 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbd2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbd2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34796/0 (socket says 172.21.15.16:34796) 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2337899930 learned_addr learned my addr 172.21.15.16:0/2337899930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbd7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2337899930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 msgr2=0x5607efbd7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbd7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2337899930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 msgr2=0x5607efbda250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 0x5607efbda250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2337899930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607ef846210 con 0x5607efbd2260 2024-04-25T01:28:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbd7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56caffd640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f56c40a2e20 0x7f56c40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.290+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbd2680 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0108c0 tx=0x7f56cc010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dc62a6f630546d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.291+0000 7f56caffd640 1 -- 172.21.15.16:0/3578211618 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f56d00cf190 con 0x7f56d8102260 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.291+0000 7f56de1d4640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f56c40a2e20 0x7f56c40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.291+0000 7f56c3fff640 1 -- 172.21.15.16:0/2337899930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc016070 con 0x5607efbd2260 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.291+0000 7f56c3fff640 1 -- 172.21.15.16:0/2337899930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56cc01b460 con 0x5607efbd2260 2024-04-25T01:28:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.291+0000 7f56c3fff640 1 -- 172.21.15.16:0/2337899930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0126b0 con 0x5607efbd2260 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 -- 172.21.15.16:0/2337899930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 msgr2=0x5607efbd2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbd2680 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0108c0 tx=0x7f56cc010d80 comp rx=0 tx=0).stop 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56de1d4640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f56c40a2e20 0x7f56c40a52e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f56d00a40b0 tx=0x7f56d00974f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 -- 172.21.15.16:0/2337899930 shutdown_connections 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 0x5607efbda250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbd7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2337899930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbd2680 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 -- 172.21.15.16:0/2337899930 >> 172.21.15.16:0/2337899930 conn(0x5607efbc94c0 msgr2=0x5607efbc98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.292+0000 7f56e2a17740 1 -- 172.21.15.16:0/2337899930 shutdown_connections 2024-04-25T01:28:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.293+0000 7f56e2a17740 1 -- 172.21.15.16:0/2337899930 wait complete. 2024-04-25T01:28:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.293+0000 7f56e2a17740 1 -- start start 2024-04-25T01:28:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.293+0000 7f56e2a17740 1 Processor -- start 2024-04-25T01:28:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbdc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd2c50 0x5607efbdcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd7dc0 0x5607efbe3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5607efbdadc0 con 0x5607efbd2260 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5607efaedb50 con 0x5607efbd2c50 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbdc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5607efbdaf40 con 0x5607efbd7dc0 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbdc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34798/0 (socket says 172.21.15.16:34798) 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56dd1d2640 1 -- 172.21.15.16:0/3894655162 learned_addr learned my addr 172.21.15.16:0/3894655162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.294+0000 7f56e2a17740 10 client.0 _ll_register_callbacks cb 0x5607efb28ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56dd1d2640 1 -- 172.21.15.16:0/3894655162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd7dc0 msgr2=0x5607efbe3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd7dc0 0x5607efbe3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56dd1d2640 1 -- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd2c50 msgr2=0x5607efbdcbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd2c50 0x5607efbdcbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56dd1d2640 1 -- 172.21.15.16:0/3894655162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607ef912620 con 0x5607efbd2260 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbdc680 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f56cc004fd0 tx=0x7f56cc005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8e403faa011d96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56c17fa640 1 -- 172.21.15.16:0/3894655162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc016070 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56c17fa640 1 -- 172.21.15.16:0/3894655162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f56cc00eb90 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56cc01b460 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56c17fa640 1 -- 172.21.15.16:0/3894655162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f56cc02d070 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56e2a17740 10 client.15399 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.295+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5607efbe6f50 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56c17fa640 1 -- 172.21.15.16:0/3894655162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56cc01b460 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56c17fa640 1 -- 172.21.15.16:0/3894655162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f56cc00f040 con 0x5607efbd2260 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56e2a17740 10 client.15399 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56e2a17740 10 client.15399 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56e2a17740 20 client.15399 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56e2a17740 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efbe83a0 0x5607efbea820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56e2a17740 4 client.15399 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607efb3f1f0 con 0x5607efbe83a0 2024-04-25T01:28:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.296+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efbe83a0 0x5607efbea820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.297+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efbe83a0 0x5607efbea820 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12e8a97c05fbc6e4 server_cookie=690642f79015b604 in_seq=0 out_seq=0 2024-04-25T01:28:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.297+0000 7f56c17fa640 10 client.15399 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56c17fa640 1 -- 172.21.15.16:0/3894655162 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5607efb3f1f0 con 0x5607efbe83a0 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56c17fa640 10 client.15399 handle_command_reply: tid=0 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 client.15399 shutdown 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 msgr2=0x5607efbdc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbdc680 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f56cc004fd0 tx=0x7f56cc005c30 comp rx=0 tx=0).stop 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 shutdown_connections 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd7dc0 0x5607efbe3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efbe83a0 0x5607efbea820 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd2c50 0x5607efbdcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 --2- 172.21.15.16:0/3894655162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2260 0x5607efbdc680 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56c17fa640 0 client.15399 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 >> 172.21.15.16:0/3894655162 conn(0x5607efbc94c0 msgr2=0x5607efbc85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 shutdown_connections 2024-04-25T01:28:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 1 -- 172.21.15.16:0/3894655162 wait complete. 2024-04-25T01:28:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.298+0000 7f56e2a17740 20 client.15399 trim_cache size 0 max 16384 2024-04-25T01:28:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 Processor -- start 2024-04-25T01:28:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 -- start start 2024-04-25T01:28:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbdc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 0x5607efbdc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 0x5607efbc7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc010df0 con 0x5607efbd7dc0 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc00e6c0 con 0x5607efbdcd10 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56e2a17740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc00c510 con 0x5607efbd2c50 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd9d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 0x5607efbdc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 0x5607efbc7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 0x5607efbc7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46612/0 (socket says 172.21.15.16:46612) 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2550092814 learned_addr learned my addr 172.21.15.16:0/2550092814 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd9d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 0x5607efbdc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34808/0 (socket says 172.21.15.16:34808) 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbdc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2550092814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 msgr2=0x5607efbdc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.321+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbdc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2550092814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 msgr2=0x5607efbdc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 0x5607efbdc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56dd1d2640 1 -- 172.21.15.16:0/2550092814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607efbdae80 con 0x5607efbdcd10 2024-04-25T01:28:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 0x5607efbdc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 0x5607efbc7d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56cc029520 tx=0x7f56cc010d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3bf9c946285b3ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56c37fe640 1 -- 172.21.15.16:0/2550092814 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc016070 con 0x5607efbdcd10 2024-04-25T01:28:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56c37fe640 1 -- 172.21.15.16:0/2550092814 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56cc022030 con 0x5607efbdcd10 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.322+0000 7f56c37fe640 1 -- 172.21.15.16:0/2550092814 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00edf0 con 0x5607efbdcd10 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 -- 172.21.15.16:0/2550092814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 msgr2=0x5607efbc7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 0x5607efbc7d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56cc029520 tx=0x7f56cc010d80 comp rx=0 tx=0).stop 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 -- 172.21.15.16:0/2550092814 shutdown_connections 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbdcd10 0x5607efbc7d30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd7dc0 0x5607efbdc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 --2- 172.21.15.16:0/2550092814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbd2c50 0x5607efbdc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 -- 172.21.15.16:0/2550092814 >> 172.21.15.16:0/2550092814 conn(0x5607efbc94c0 msgr2=0x5607efbc98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 -- 172.21.15.16:0/2550092814 shutdown_connections 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.323+0000 7f56e2a17740 1 -- 172.21.15.16:0/2550092814 wait complete. 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 -- start start 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 Processor -- start 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 0x5607efbe0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 0x5607efbe11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbdcd10 0x5607efbe21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc00cb80 con 0x5607efbd2c50 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc00dff0 con 0x5607efbd7dc0 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc00d250 con 0x5607efbdcd10 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.324+0000 7f56e2a17740 10 client.0 _ll_register_callbacks cb 0x7f56cc0107a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 0x5607efbe0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 0x5607efbe0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34818/0 (socket says 172.21.15.16:34818) 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 -- 172.21.15.16:0/33563264 learned_addr learned my addr 172.21.15.16:0/33563264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbdcd10 0x5607efbe21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 -- 172.21.15.16:0/33563264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbdcd10 msgr2=0x5607efbe21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbdcd10 0x5607efbe21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 -- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 msgr2=0x5607efbe11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 0x5607efbe11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 -- 172.21.15.16:0/33563264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607efbe27c0 con 0x5607efbd2c50 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56dd1d2640 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 0x5607efbe0c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009e90 tx=0x7f56cc010850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487675dc908f382a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.325+0000 7f56c17fa640 1 -- 172.21.15.16:0/33563264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc037040 con 0x5607efbd2c50 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56cc022030 con 0x5607efbd2c50 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56c17fa640 1 -- 172.21.15.16:0/33563264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f56cc00eb70 con 0x5607efbd2c50 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56c17fa640 1 -- 172.21.15.16:0/33563264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f56cc016040 con 0x5607efbd2c50 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 10 client.15405 fetch_fsmap learned FSMap version 22 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5607efc4ee10 con 0x5607efbd2c50 2024-04-25T01:28:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56c17fa640 1 -- 172.21.15.16:0/33563264 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56cc022030 con 0x5607efbd2c50 2024-04-25T01:28:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56c17fa640 1 -- 172.21.15.16:0/33563264 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f56cc03e070 con 0x5607efbd2c50 2024-04-25T01:28:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 10 client.15405 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:28:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 10 client.15405 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:28:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 20 client.15405 populate_metadata read hostname 'smithi016' 2024-04-25T01:28:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efc4f6f0 0x5607efc51b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 4 client.15405 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:28:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.326+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5607efb39540 con 0x5607efc4f6f0 2024-04-25T01:28:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.327+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efc4f6f0 0x5607efc51b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.327+0000 7f56c17fa640 10 client.15405 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.327+0000 7f56dd9d3640 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efc4f6f0 0x5607efc51b10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d78f74b9383617b9 server_cookie=8642098219f69ad6 in_seq=0 out_seq=0 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56c17fa640 1 -- 172.21.15.16:0/33563264 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5607efb39540 con 0x5607efc4f6f0 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56c17fa640 10 client.15405 handle_command_reply: tid=0 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 client.15405 shutdown 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 msgr2=0x5607efbe0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 0x5607efbe0c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009e90 tx=0x7f56cc010850 comp rx=0 tx=0).stop 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 shutdown_connections 2024-04-25T01:28:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607efbdcd10 0x5607efbe21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5607efc4f6f0 0x5607efc51b10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5607efbd7dc0 0x5607efbe11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 --2- 172.21.15.16:0/33563264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5607efbd2c50 0x5607efbe0c80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56c17fa640 0 client.15405 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.337+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 >> 172.21.15.16:0/33563264 conn(0x5607efbc94c0 msgr2=0x5607efbd4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 shutdown_connections 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56e2a17740 1 -- 172.21.15.16:0/33563264 wait complete. 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56e2a17740 20 client.15405 trim_cache size 0 max 16384 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:28:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:28:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:28:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:28:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f56c40a2e20 msgr2=0x7f56c40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f56c40a2e20 0x7f56c40a52e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f56d00a40b0 tx=0x7f56d00974f0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 msgr2=0x7f56d819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.338+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 0x7f56d819ef50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f56d000e9b0 tx=0x7f56d000ee70 comp rx=0 tx=0).stop 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 shutdown_connections 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f56c40a2e20 0x7f56c40a52e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56d8102260 0x7f56d819ef50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56d81018a0 0x7f56d8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 --2- 172.21.15.16:0/3578211618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d81006b0 0x7f56d81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 >> 172.21.15.16:0/3578211618 conn(0x7f56d80fbc20 msgr2=0x7f56d80fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 shutdown_connections 2024-04-25T01:28:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:28:54.339+0000 7f56dfc37640 1 -- 172.21.15.16:0/3578211618 wait complete. 2024-04-25T01:28:55.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:55 smithi016 ceph-mon[29153]: pgmap v404: 97 pgs: 97 active+clean; 2.8 GiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 184 MiB/s wr, 105 op/s 2024-04-25T01:28:55.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:55 smithi038 ceph-mon[32841]: pgmap v404: 97 pgs: 97 active+clean; 2.8 GiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 184 MiB/s wr, 105 op/s 2024-04-25T01:28:55.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:55 smithi156 ceph-mon[32631]: pgmap v404: 97 pgs: 97 active+clean; 2.8 GiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 184 MiB/s wr, 105 op/s 2024-04-25T01:28:57.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:57 smithi016 ceph-mon[29153]: pgmap v405: 97 pgs: 97 active+clean; 2.9 GiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 146 MiB/s wr, 84 op/s 2024-04-25T01:28:57.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:57 smithi038 ceph-mon[32841]: pgmap v405: 97 pgs: 97 active+clean; 2.9 GiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 146 MiB/s wr, 84 op/s 2024-04-25T01:28:57.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:57 smithi156 ceph-mon[32631]: pgmap v405: 97 pgs: 97 active+clean; 2.9 GiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 146 MiB/s wr, 84 op/s 2024-04-25T01:28:59.609 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:28:59 smithi016 ceph-mon[29153]: pgmap v406: 97 pgs: 97 active+clean; 3.3 GiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 172 MiB/s wr, 94 op/s 2024-04-25T01:28:59.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:28:59 smithi038 ceph-mon[32841]: pgmap v406: 97 pgs: 97 active+clean; 3.3 GiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 172 MiB/s wr, 94 op/s 2024-04-25T01:28:59.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:28:59 smithi156 ceph-mon[32631]: pgmap v406: 97 pgs: 97 active+clean; 3.3 GiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 172 MiB/s wr, 94 op/s 2024-04-25T01:29:01.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:01 smithi016 ceph-mon[29153]: pgmap v407: 97 pgs: 97 active+clean; 3.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 169 MiB/s wr, 94 op/s 2024-04-25T01:29:01.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:01 smithi038 ceph-mon[32841]: pgmap v407: 97 pgs: 97 active+clean; 3.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 169 MiB/s wr, 94 op/s 2024-04-25T01:29:01.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:01 smithi156 ceph-mon[32631]: pgmap v407: 97 pgs: 97 active+clean; 3.7 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 169 MiB/s wr, 94 op/s 2024-04-25T01:29:03.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:03 smithi016 ceph-mon[29153]: pgmap v408: 97 pgs: 97 active+clean; 3.9 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 164 MiB/s wr, 92 op/s 2024-04-25T01:29:03.717 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:03 smithi038 ceph-mon[32841]: pgmap v408: 97 pgs: 97 active+clean; 3.9 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 164 MiB/s wr, 92 op/s 2024-04-25T01:29:03.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:03 smithi156 ceph-mon[32631]: pgmap v408: 97 pgs: 97 active+clean; 3.9 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 164 MiB/s wr, 92 op/s 2024-04-25T01:29:05.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:05 smithi038 ceph-mon[32841]: pgmap v409: 97 pgs: 97 active+clean; 4.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 184 MiB/s wr, 100 op/s 2024-04-25T01:29:05.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:05 smithi156 ceph-mon[32631]: pgmap v409: 97 pgs: 97 active+clean; 4.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 184 MiB/s wr, 100 op/s 2024-04-25T01:29:05.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:05 smithi016 ceph-mon[29153]: pgmap v409: 97 pgs: 97 active+clean; 4.5 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 184 MiB/s wr, 100 op/s 2024-04-25T01:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 -- 172.21.15.16:0/2155293567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 msgr2=0x7f87ac100420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 --2- 172.21.15.16:0/2155293567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac100420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f879c009860 tx=0x7f879c02a340 comp rx=0 tx=0).stop 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 -- 172.21.15.16:0/2155293567 shutdown_connections 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 --2- 172.21.15.16:0/2155293567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac101970 0x7f87ac108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 --2- 172.21.15.16:0/2155293567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac100fb0 0x7f87ac101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 --2- 172.21.15.16:0/2155293567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac100420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 -- 172.21.15.16:0/2155293567 >> 172.21.15.16:0/2155293567 conn(0x7f87ac0faf60 msgr2=0x7f87ac0fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 -- 172.21.15.16:0/2155293567 shutdown_connections 2024-04-25T01:29:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.478+0000 7f87b2288640 1 -- 172.21.15.16:0/2155293567 wait complete. 2024-04-25T01:29:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.479+0000 7f87b2288640 1 Processor -- start 2024-04-25T01:29:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.479+0000 7f87b2288640 1 -- start start 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b2288640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac1956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b2288640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac100fb0 0x7f87ac195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b2288640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac101970 0x7f87ac19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b2288640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac10b0a0 con 0x7f87ac101970 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b2288640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac10af20 con 0x7f87ac100000 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b2288640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac10b220 con 0x7f87ac100fb0 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac1956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac1956c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:40352/0 (socket says 172.21.15.16:40352) 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 -- 172.21.15.16:0/3830762084 learned_addr learned my addr 172.21.15.16:0/3830762084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87b0825640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac101970 0x7f87ac19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac100fb0 0x7f87ac195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 -- 172.21.15.16:0/3830762084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac100fb0 msgr2=0x7f87ac195c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac100fb0 0x7f87ac195c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 -- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac101970 msgr2=0x7f87ac19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac101970 0x7f87ac19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87abfff640 1 -- 172.21.15.16:0/3830762084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac19d2e0 con 0x7f87ac100000 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.480+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac100fb0 0x7f87ac195c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.481+0000 7f87abfff640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac1956c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f879c02a860 tx=0x7f879c033d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.481+0000 7f87a97fa640 1 -- 172.21.15.16:0/3830762084 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c042070 con 0x7f87ac100000 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.481+0000 7f87a97fa640 1 -- 172.21.15.16:0/3830762084 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f879c0054e0 con 0x7f87ac100000 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.481+0000 7f87a97fa640 1 -- 172.21.15.16:0/3830762084 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c037040 con 0x7f87ac100000 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.481+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ac19d570 con 0x7f87ac100000 2024-04-25T01:29:07.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.481+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ac19da10 con 0x7f87ac100000 2024-04-25T01:29:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87a97fa640 1 -- 172.21.15.16:0/3830762084 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f879c045070 con 0x7f87ac100000 2024-04-25T01:29:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 Processor -- start 2024-04-25T01:29:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 -- start start 2024-04-25T01:29:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba33ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba386e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba38d70 0x56384ba3b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56384b992cb0 con 0x56384ba38d70 2024-04-25T01:29:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56384b93b5b0 con 0x56384ba34170 2024-04-25T01:29:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.484+0000 7f87b4f90740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56384b98ec70 con 0x56384ba33780 2024-04-25T01:29:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87a97fa640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f87940a2cb0 0x7f87940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba386e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:40366/0 (socket says 172.21.15.16:40366) 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba38d70 0x56384ba3b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87abfff640 1 -- 172.21.15.16:0/2787107742 learned_addr learned my addr 172.21.15.16:0/2787107742 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2787107742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 msgr2=0x56384ba33ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba33ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87b0825640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f87940a2cb0 0x7f87940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2787107742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 msgr2=0x56384ba386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2787107742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384b6a7210 con 0x56384ba38d70 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.485+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba38d70 0x56384ba3b230 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0ffd20 tx=0x7f87a000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ed9017556ac208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.486+0000 7f87927fc640 1 -- 172.21.15.16:0/2787107742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0005020 con 0x56384ba38d70 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.486+0000 7f87a97fa640 1 -- 172.21.15.16:0/3830762084 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f879c0ee9a0 con 0x7f87ac100000 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.486+0000 7f87927fc640 1 -- 172.21.15.16:0/2787107742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87a00047a0 con 0x56384ba38d70 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.486+0000 7f87927fc640 1 -- 172.21.15.16:0/2787107742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a000c810 con 0x56384ba38d70 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 -- 172.21.15.16:0/2787107742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba38d70 msgr2=0x56384ba3b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba38d70 0x56384ba3b230 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0ffd20 tx=0x7f87a000e820 comp rx=0 tx=0).stop 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b0825640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f87940a2cb0 0x7f87940a5170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f87a4005fd0 tx=0x7f87a4009bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 -- 172.21.15.16:0/2787107742 shutdown_connections 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba38d70 0x56384ba3b230 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2787107742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba33ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 -- 172.21.15.16:0/2787107742 >> 172.21.15.16:0/2787107742 conn(0x56384ba2ba00 msgr2=0x56384ba2a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 -- 172.21.15.16:0/2787107742 shutdown_connections 2024-04-25T01:29:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 -- 172.21.15.16:0/2787107742 wait complete. 2024-04-25T01:29:07.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.487+0000 7f87b4f90740 1 -- start start 2024-04-25T01:29:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 Processor -- start 2024-04-25T01:29:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba2f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba2ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba2c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba2f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56384ba3bd50 con 0x56384ba33780 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba2f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45796/0 (socket says 172.21.15.16:45796) 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87abfff640 1 -- 172.21.15.16:0/111887568 learned_addr learned my addr 172.21.15.16:0/111887568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56384b94eb50 con 0x56384ba34170 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.488+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56384ba3bed0 con 0x56384ba38d70 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87b4f90740 10 client.0 _ll_register_callbacks cb 0x56384b989ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba2ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87b0825640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba2c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87abfff640 1 -- 172.21.15.16:0/111887568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 msgr2=0x56384ba2c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87abfff640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba2c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87abfff640 1 -- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 msgr2=0x56384ba2ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87abfff640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba2ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87abfff640 1 -- 172.21.15.16:0/111887568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384b773620 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f87abfff640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba2f9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f879c0b7640 tx=0x7f879c009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e45f596ecfadbd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.489+0000 7f878b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.490+0000 7f878b7fe640 1 -- 172.21.15.16:0/111887568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c0bad20 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.490+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a00047a0 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.490+0000 7f878b7fe640 1 -- 172.21.15.16:0/111887568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f879c0db0d0 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.490+0000 7f878b7fe640 1 -- 172.21.15.16:0/111887568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f879c0bab40 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.490+0000 7f87b4f90740 10 client.15417 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.490+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56384ba4c3c0 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f878b7fe640 1 -- 172.21.15.16:0/111887568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87a00047a0 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f878b7fe640 1 -- 172.21.15.16:0/111887568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f879c03a070 con 0x56384ba33780 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87b4f90740 10 client.15417 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87b4f90740 10 client.15417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87b4f90740 20 client.15417 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87b4f90740 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384ba4ddc0 0x56384ba50240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87b4f90740 4 client.15417 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56384b9a01f0 con 0x56384ba4ddc0 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384ba4ddc0 0x56384ba50240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f878b7fe640 10 client.15417 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.491+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384ba4ddc0 0x56384ba50240 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59caccac3b5cd211 server_cookie=726bff1956c3877c in_seq=0 out_seq=0 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.492+0000 7f878b7fe640 1 -- 172.21.15.16:0/111887568 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56384b9a01f0 con 0x56384ba4ddc0 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.492+0000 7f878b7fe640 10 client.15417 handle_command_reply: tid=0 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.492+0000 7f87b4f90740 1 client.15417 shutdown 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.492+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 msgr2=0x56384ba2f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.492+0000 7f87b4f90740 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba2f9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f879c0b7640 tx=0x7f879c009550 comp rx=0 tx=0).stop 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 shutdown_connections 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba2c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384ba4ddc0 0x56384ba50240 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba2ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 --2- 172.21.15.16:0/111887568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba2f9f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 >> 172.21.15.16:0/111887568 conn(0x56384ba2ba00 msgr2=0x56384ba295c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f878b7fe640 0 client.15417 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 shutdown_connections 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 1 -- 172.21.15.16:0/111887568 wait complete. 2024-04-25T01:29:07.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.493+0000 7f87b4f90740 20 client.15417 trim_cache size 0 max 16384 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 Processor -- start 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 -- start start 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba3ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 0x56384ba3b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba38d70 0x56384ba39150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac1a91b0 con 0x56384ba34170 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac1a6a80 con 0x56384ba38d70 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b4f90740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac1a48d0 con 0x56384ba33780 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 0x56384ba3b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87ab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 0x56384ba3b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45806/0 (socket says 172.21.15.16:45806) 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2389404926 learned_addr learned my addr 172.21.15.16:0/2389404926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87abfff640 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba3ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.519+0000 7f87b0825640 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba38d70 0x56384ba39150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2389404926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 msgr2=0x56384ba3ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba3ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2389404926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba38d70 msgr2=0x56384ba39150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba38d70 0x56384ba39150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f87ab7fe640 1 -- 172.21.15.16:0/2389404926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384ba2d120 con 0x56384ba34170 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 0x56384ba3b0b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f87a0005fd0 tx=0x7f87a0005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cad14fc9c1795 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f8791ffb640 1 -- 172.21.15.16:0/2389404926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a000a8c0 con 0x56384ba34170 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f8791ffb640 1 -- 172.21.15.16:0/2389404926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87a00044f0 con 0x56384ba34170 2024-04-25T01:29:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.520+0000 7f8791ffb640 1 -- 172.21.15.16:0/2389404926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a00035f0 con 0x56384ba34170 2024-04-25T01:29:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 -- 172.21.15.16:0/2389404926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 msgr2=0x56384ba3b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 0x56384ba3b0b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f87a0005fd0 tx=0x7f87a0005d70 comp rx=0 tx=0).stop 2024-04-25T01:29:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 -- 172.21.15.16:0/2389404926 shutdown_connections 2024-04-25T01:29:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba38d70 0x56384ba39150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba34170 0x56384ba3b0b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 --2- 172.21.15.16:0/2389404926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba33780 0x56384ba3ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 -- 172.21.15.16:0/2389404926 >> 172.21.15.16:0/2389404926 conn(0x56384ba2ba00 msgr2=0x56384ba32fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 -- 172.21.15.16:0/2389404926 shutdown_connections 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 -- 172.21.15.16:0/2389404926 wait complete. 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.521+0000 7f87b4f90740 1 -- start start 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 Processor -- start 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba548f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba54e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba55370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac1a4f40 con 0x56384ba33780 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac1a63b0 con 0x56384ba34170 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac1a5610 con 0x56384ba38d70 2024-04-25T01:29:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b4f90740 10 client.0 _ll_register_callbacks cb 0x7f879c0f4240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45816/0 (socket says 172.21.15.16:45816) 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87abfff640 1 -- 172.21.15.16:0/1354095156 learned_addr learned my addr 172.21.15.16:0/1354095156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87b0825640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba55370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.522+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba54e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87abfff640 1 -- 172.21.15.16:0/1354095156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 msgr2=0x56384ba55370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87abfff640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba55370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87abfff640 1 -- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 msgr2=0x56384ba54e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87abfff640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba54e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87abfff640 1 -- 172.21.15.16:0/1354095156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384bab2420 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87b0825640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba55370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba54e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87abfff640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba548f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f879c0b7b60 tx=0x7f879c0c0950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1883cf608964467 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f878b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f878b7fe640 1 -- 172.21.15.16:0/1354095156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c0bad20 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f878b7fe640 1 -- 172.21.15.16:0/1354095156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f879c0c39c0 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a00044f0 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f878b7fe640 1 -- 172.21.15.16:0/1354095156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f879c0db2e0 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87b4f90740 10 client.15423 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.523+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56384bab3730 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f878b7fe640 1 -- 172.21.15.16:0/1354095156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f879c09d020 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f878b7fe640 1 -- 172.21.15.16:0/1354095156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f879c03a070 con 0x56384ba33780 2024-04-25T01:29:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87b4f90740 10 client.15423 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87b4f90740 10 client.15423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87b4f90740 20 client.15423 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87b4f90740 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384bab3f50 0x56384bab6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87b4f90740 4 client.15423 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56384b99a540 con 0x56384bab3f50 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.524+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384bab3f50 0x56384bab6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.525+0000 7f878b7fe640 10 client.15423 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.525+0000 7f87ab7fe640 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384bab3f50 0x56384bab6370 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a878943e66b98f91 server_cookie=f866121408c481d in_seq=0 out_seq=0 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f878b7fe640 1 -- 172.21.15.16:0/1354095156 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56384b99a540 con 0x56384bab3f50 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f878b7fe640 10 client.15423 handle_command_reply: tid=0 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 client.15423 shutdown 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 msgr2=0x56384ba548f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba548f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f879c0b7b60 tx=0x7f879c0c0950 comp rx=0 tx=0).stop 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 shutdown_connections 2024-04-25T01:29:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56384ba38d70 0x56384ba55370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56384bab3f50 0x56384bab6370 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56384ba34170 0x56384ba54e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 --2- 172.21.15.16:0/1354095156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56384ba33780 0x56384ba548f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f878b7fe640 0 client.15423 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 >> 172.21.15.16:0/1354095156 conn(0x56384ba2ba00 msgr2=0x56384ba2f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 shutdown_connections 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 1 -- 172.21.15.16:0/1354095156 wait complete. 2024-04-25T01:29:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.536+0000 7f87b4f90740 20 client.15423 trim_cache size 0 max 16384 2024-04-25T01:29:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:29:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:29:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:29:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:29:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f87940a2cb0 msgr2=0x7f87940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f87940a2cb0 0x7f87940a5170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f87a4005fd0 tx=0x7f87a4009bf0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 msgr2=0x7f87ac1956c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac1956c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f879c02a860 tx=0x7f879c033d60 comp rx=0 tx=0).stop 2024-04-25T01:29:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 shutdown_connections 2024-04-25T01:29:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f87940a2cb0 0x7f87940a5170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac101970 0x7f87ac19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac100fb0 0x7f87ac195c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 --2- 172.21.15.16:0/3830762084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87ac100000 0x7f87ac1956c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 >> 172.21.15.16:0/3830762084 conn(0x7f87ac0faf60 msgr2=0x7f87ac104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 shutdown_connections 2024-04-25T01:29:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:07.537+0000 7f87b2288640 1 -- 172.21.15.16:0/3830762084 wait complete. 2024-04-25T01:29:07.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:07 smithi038 ceph-mon[32841]: pgmap v410: 97 pgs: 97 active+clean; 4.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 148 MiB/s wr, 85 op/s 2024-04-25T01:29:07.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:07 smithi156 ceph-mon[32631]: pgmap v410: 97 pgs: 97 active+clean; 4.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 148 MiB/s wr, 85 op/s 2024-04-25T01:29:07.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:07 smithi016 ceph-mon[29153]: pgmap v410: 97 pgs: 97 active+clean; 4.6 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 148 MiB/s wr, 85 op/s 2024-04-25T01:29:09.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:09 smithi038 ceph-mon[32841]: pgmap v411: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 173 MiB/s wr, 94 op/s 2024-04-25T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:09 smithi156 ceph-mon[32631]: pgmap v411: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 173 MiB/s wr, 94 op/s 2024-04-25T01:29:09.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:09 smithi016 ceph-mon[29153]: pgmap v411: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 173 MiB/s wr, 94 op/s 2024-04-25T01:29:11.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:11 smithi038 ceph-mon[32841]: pgmap v412: 97 pgs: 97 active+clean; 5.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 MiB/s wr, 94 op/s 2024-04-25T01:29:11.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:11 smithi156 ceph-mon[32631]: pgmap v412: 97 pgs: 97 active+clean; 5.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 MiB/s wr, 94 op/s 2024-04-25T01:29:11.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:11 smithi016 ceph-mon[29153]: pgmap v412: 97 pgs: 97 active+clean; 5.3 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 MiB/s wr, 94 op/s 2024-04-25T01:29:13.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:13 smithi038 ceph-mon[32841]: pgmap v413: 97 pgs: 97 active+clean; 5.6 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 162 MiB/s wr, 90 op/s 2024-04-25T01:29:13.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:13 smithi156 ceph-mon[32631]: pgmap v413: 97 pgs: 97 active+clean; 5.6 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 162 MiB/s wr, 90 op/s 2024-04-25T01:29:13.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:13 smithi016 ceph-mon[29153]: pgmap v413: 97 pgs: 97 active+clean; 5.6 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 162 MiB/s wr, 90 op/s 2024-04-25T01:29:15.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:15 smithi038 ceph-mon[32841]: pgmap v414: 97 pgs: 97 active+clean; 6.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 183 MiB/s wr, 100 op/s 2024-04-25T01:29:15.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:15 smithi156 ceph-mon[32631]: pgmap v414: 97 pgs: 97 active+clean; 6.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 183 MiB/s wr, 100 op/s 2024-04-25T01:29:15.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:15 smithi016 ceph-mon[29153]: pgmap v414: 97 pgs: 97 active+clean; 6.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 183 MiB/s wr, 100 op/s 2024-04-25T01:29:17.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:17 smithi038 ceph-mon[32841]: pgmap v415: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 147 MiB/s wr, 84 op/s 2024-04-25T01:29:17.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:17 smithi156 ceph-mon[32631]: pgmap v415: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 147 MiB/s wr, 84 op/s 2024-04-25T01:29:17.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:17 smithi016 ceph-mon[29153]: pgmap v415: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 147 MiB/s wr, 84 op/s 2024-04-25T01:29:19.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:19 smithi038 ceph-mon[32841]: pgmap v416: 97 pgs: 97 active+clean; 6.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 174 MiB/s wr, 95 op/s 2024-04-25T01:29:19.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:19 smithi156 ceph-mon[32631]: pgmap v416: 97 pgs: 97 active+clean; 6.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 174 MiB/s wr, 95 op/s 2024-04-25T01:29:19.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:19 smithi016 ceph-mon[29153]: pgmap v416: 97 pgs: 97 active+clean; 6.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 174 MiB/s wr, 95 op/s 2024-04-25T01:29:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.664+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/2681043733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c741018d0 msgr2=0x7f4c74101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.664+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/2681043733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c741018d0 0x7f4c74101d50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009900 tx=0x7f4c6802a410 comp rx=0 tx=0).stop 2024-04-25T01:29:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/2681043733 shutdown_connections 2024-04-25T01:29:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/2681043733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c74102290 0x7f4c74108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/2681043733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c741018d0 0x7f4c74101d50 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/2681043733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741006e0 0x7f4c74100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/2681043733 >> 172.21.15.16:0/2681043733 conn(0x7f4c740fbc50 msgr2=0x7f4c740fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/2681043733 shutdown_connections 2024-04-25T01:29:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.665+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/2681043733 wait complete. 2024-04-25T01:29:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 Processor -- start 2024-04-25T01:29:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 -- start start 2024-04-25T01:29:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c741006e0 0x7f4c74196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 0x7f4c74196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c74102290 0x7f4c7419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410c260 con 0x7f4c74102290 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410c0e0 con 0x7f4c741006e0 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.666+0000 7f4c7a0a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410c3e0 con 0x7f4c741018d0 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c72ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 0x7f4c74196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c74102290 0x7f4c7419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c741006e0 0x7f4c74196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c72ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 0x7f4c74196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:57708/0 (socket says 172.21.15.16:57708) 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c72ffd640 1 -- 172.21.15.16:0/865870532 learned_addr learned my addr 172.21.15.16:0/865870532 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 -- 172.21.15.16:0/865870532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 msgr2=0x7f4c74196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 0x7f4c74196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 -- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c74102290 msgr2=0x7f4c7419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c74102290 0x7f4c7419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 -- 172.21.15.16:0/865870532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c74069a40 con 0x7f4c741006e0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 0x7f4c74196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.667+0000 7f4c737fe640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c741006e0 0x7f4c74196850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4c74101730 tx=0x7f4c6400dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.668+0000 7f4c70ff9640 1 -- 172.21.15.16:0/865870532 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64017070 con 0x7f4c741006e0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.668+0000 7f4c70ff9640 1 -- 172.21.15.16:0/865870532 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c64010460 con 0x7f4c741006e0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.668+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c74069d30 con 0x7f4c741006e0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.668+0000 7f4c70ff9640 1 -- 172.21.15.16:0/865870532 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6400f600 con 0x7f4c741006e0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.668+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7406a230 con 0x7f4c741006e0 2024-04-25T01:29:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.670+0000 7f4c70ff9640 1 -- 172.21.15.16:0/865870532 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4c6401a070 con 0x7f4c741006e0 2024-04-25T01:29:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.671+0000 7f4c70ff9640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4c5c0a2c30 0x7f4c5c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.671+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4c5c0a2c30 0x7f4c5c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.671+0000 7f4c70ff9640 1 -- 172.21.15.16:0/865870532 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4c640ce970 con 0x7f4c741006e0 2024-04-25T01:29:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.671+0000 7f4c7cdb4740 1 Processor -- start 2024-04-25T01:29:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 -- start start 2024-04-25T01:29:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f99d127cb0 con 0x55f99d1d21c0 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f99d0d05b0 con 0x55f99d1cc660 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f99d123c70 con 0x55f99d1cd050 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4c5c0a2c30 0x7f4c5c0a50f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4c74197d60 tx=0x7f4c68005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:44500/0 (socket says 172.21.15.16:44500) 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c72ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c737fe640 1 -- 172.21.15.16:0/2282327533 learned_addr learned my addr 172.21.15.16:0/2282327533 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46136/0 (socket says 172.21.15.16:46136) 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c73fff640 1 -- 172.21.15.16:0/2282327533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 msgr2=0x55f99d1d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.672+0000 7f4c73fff640 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c73fff640 1 -- 172.21.15.16:0/2282327533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 msgr2=0x55f99d1cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c73fff640 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c73fff640 1 -- 172.21.15.16:0/2282327533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f99ce3c210 con 0x55f99d1d21c0 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c73fff640 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d4650 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009f30 tx=0x7f4c6c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b3153f35a5b3d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c59ffb640 1 -- 172.21.15.16:0/2282327533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c016070 con 0x55f99d1d21c0 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c59ffb640 1 -- 172.21.15.16:0/2282327533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c6c004d20 con 0x55f99d1d21c0 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.673+0000 7f4c59ffb640 1 -- 172.21.15.16:0/2282327533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c012740 con 0x55f99d1d21c0 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2282327533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 msgr2=0x55f99d1d4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d4650 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009f30 tx=0x7f4c6c010800 comp rx=0 tx=0).stop 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2282327533 shutdown_connections 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d4650 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2282327533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2282327533 >> 172.21.15.16:0/2282327533 conn(0x55f99d1c7cd0 msgr2=0x55f99d1c80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2282327533 shutdown_connections 2024-04-25T01:29:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.674+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2282327533 wait complete. 2024-04-25T01:29:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 -- start start 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 Processor -- start 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 0x55f99d1d6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1d21c0 0x55f99d1de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f99d1d5170 con 0x55f99d1cd050 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f99d0e3b50 con 0x55f99d1cc660 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f99d1d52f0 con 0x55f99d1d21c0 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c7cdb4740 10 client.0 _ll_register_callbacks cb 0x55f99d11eca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 0x55f99d1d6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c72ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 0x55f99d1d6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46148/0 (socket says 172.21.15.16:46148) 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c72ffd640 1 -- 172.21.15.16:0/2992364699 learned_addr learned my addr 172.21.15.16:0/2992364699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c73fff640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1d21c0 0x55f99d1de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.675+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c72ffd640 1 -- 172.21.15.16:0/2992364699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1d21c0 msgr2=0x55f99d1de000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1d21c0 0x55f99d1de000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c72ffd640 1 -- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 msgr2=0x55f99d1d6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1d6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c72ffd640 1 -- 172.21.15.16:0/2992364699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f99cf08620 con 0x55f99d1cd050 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c73fff640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1d21c0 0x55f99d1de000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 0x55f99d1d6fc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4c68005ab0 tx=0x7f4c6802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aade2fd716d1bdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:29:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c52ffd640 1 -- 172.21.15.16:0/2992364699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68035050 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c52ffd640 1 -- 172.21.15.16:0/2992364699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4c68031990 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c6c004d20 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.676+0000 7f4c52ffd640 1 -- 172.21.15.16:0/2992364699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4c68031b90 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 10 client.15435 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f99d1e13e0 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c52ffd640 1 -- 172.21.15.16:0/2992364699 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c6c004d20 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c52ffd640 1 -- 172.21.15.16:0/2992364699 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4c6803c440 con 0x55f99d1cd050 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 10 client.15435 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 10 client.15435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 20 client.15435 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d1e2b50 0x55f99d1e4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 4 client.15435 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f99d1351f0 con 0x55f99d1e2b50 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.677+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d1e2b50 0x55f99d1e4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.678+0000 7f4c52ffd640 10 client.15435 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.678+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d1e2b50 0x55f99d1e4f60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9f7f5cd8477f170 server_cookie=bcd8674ed36e18bc in_seq=0 out_seq=0 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.678+0000 7f4c52ffd640 1 -- 172.21.15.16:0/2992364699 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f99d1351f0 con 0x55f99d1e2b50 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.678+0000 7f4c52ffd640 10 client.15435 handle_command_reply: tid=0 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.678+0000 7f4c7cdb4740 1 client.15435 shutdown 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 msgr2=0x55f99d1d6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 0x55f99d1d6fc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4c68005ab0 tx=0x7f4c6802a410 comp rx=0 tx=0).stop 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 shutdown_connections 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1d21c0 0x55f99d1de000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cd050 0x55f99d1d6fc0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d1e2b50 0x55f99d1e4f60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2992364699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cc660 0x55f99d1d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c52ffd640 0 client.15435 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 >> 172.21.15.16:0/2992364699 conn(0x55f99d1c7cd0 msgr2=0x55f99d1be5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 shutdown_connections 2024-04-25T01:29:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2992364699 wait complete. 2024-04-25T01:29:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.679+0000 7f4c7cdb4740 20 client.15435 trim_cache size 0 max 16384 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 Processor -- start 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 -- start start 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 0x55f99d1c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cd050 0x55f99d1ddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741a92d0 con 0x55f99d1d21c0 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741a6ba0 con 0x55f99d1cd050 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.709+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741a49f0 con 0x55f99d1cc660 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 0x55f99d1c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 0x55f99d1c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:57736/0 (socket says 172.21.15.16:57736) 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 -- 172.21.15.16:0/2676028032 learned_addr learned my addr 172.21.15.16:0/2676028032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cd050 0x55f99d1ddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 -- 172.21.15.16:0/2676028032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cd050 msgr2=0x55f99d1ddfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cd050 0x55f99d1ddfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 -- 172.21.15.16:0/2676028032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 msgr2=0x55f99d1d6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 -- 172.21.15.16:0/2676028032 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f99d1d5430 con 0x55f99d1cc660 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c73fff640 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c737fe640 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 0x55f99d1c80b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c6409b390 tx=0x7f4c6409b0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832cca91569d1302 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c597fa640 1 -- 172.21.15.16:0/2676028032 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c640b4070 con 0x55f99d1cc660 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c597fa640 1 -- 172.21.15.16:0/2676028032 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c64099560 con 0x55f99d1cc660 2024-04-25T01:29:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.710+0000 7f4c597fa640 1 -- 172.21.15.16:0/2676028032 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64021040 con 0x55f99d1cc660 2024-04-25T01:29:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2676028032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 msgr2=0x55f99d1c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 0x55f99d1c80b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c6409b390 tx=0x7f4c6409b0b0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2676028032 shutdown_connections 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1d21c0 0x55f99d1d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1cd050 0x55f99d1ddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/2676028032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cc660 0x55f99d1c80b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.711+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2676028032 >> 172.21.15.16:0/2676028032 conn(0x55f99d1c7cd0 msgr2=0x55f99d1cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.712+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2676028032 shutdown_connections 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.712+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/2676028032 wait complete. 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.712+0000 7f4c7cdb4740 1 -- start start 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.712+0000 7f4c7cdb4740 1 Processor -- start 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cc660 0x55f99d1ddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1da810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 0x55f99d1dad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741a5060 con 0x55f99d1cc660 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741a64d0 con 0x55f99d1d21c0 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741a5730 con 0x55f99d1cd050 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c7cdb4740 10 client.0 _ll_register_callbacks cb 0x7f4c68003710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 0x55f99d1dad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cc660 0x55f99d1ddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 0x55f99d1dad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:44528/0 (socket says 172.21.15.16:44528) 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 -- 172.21.15.16:0/71003188 learned_addr learned my addr 172.21.15.16:0/71003188 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1da810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 -- 172.21.15.16:0/71003188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 msgr2=0x55f99d1da810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1da810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 -- 172.21.15.16:0/71003188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cc660 msgr2=0x55f99d1ddd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cc660 0x55f99d1ddd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 -- 172.21.15.16:0/71003188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f99d1db3f0 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c72ffd640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1da810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c73fff640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 0x55f99d1dad50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009f30 tx=0x7f4c6c003f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290fcf99f5ce7def server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.713+0000 7f4c52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c52ffd640 1 -- 172.21.15.16:0/71003188 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c016070 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c64099560 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c52ffd640 1 -- 172.21.15.16:0/71003188 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4c6c0054d0 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c52ffd640 1 -- 172.21.15.16:0/71003188 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4c6c0056f0 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 10 client.25112 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f99d248220 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c52ffd640 1 -- 172.21.15.16:0/71003188 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c64099560 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c52ffd640 1 -- 172.21.15.16:0/71003188 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4c6c00f040 con 0x55f99d1d21c0 2024-04-25T01:29:20.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 10 client.25112 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 10 client.25112 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 20 client.25112 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d248890 0x55f99d24ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 4 client.25112 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.714+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f99d12f540 con 0x55f99d248890 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.715+0000 7f4c737fe640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d248890 0x55f99d24ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.715+0000 7f4c52ffd640 10 client.25112 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.715+0000 7f4c737fe640 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d248890 0x55f99d24ac80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9b48b9e11a50159 server_cookie=4f401d9d37b13e45 in_seq=0 out_seq=0 2024-04-25T01:29:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c52ffd640 1 -- 172.21.15.16:0/71003188 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f99d12f540 con 0x55f99d248890 2024-04-25T01:29:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c52ffd640 10 client.25112 handle_command_reply: tid=0 2024-04-25T01:29:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 client.25112 shutdown 2024-04-25T01:29:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 msgr2=0x55f99d1dad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 0x55f99d1dad50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009f30 tx=0x7f4c6c003f00 comp rx=0 tx=0).stop 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 shutdown_connections 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f99d248890 0x55f99d24ac80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f99d1d21c0 0x55f99d1dad50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99d1cd050 0x55f99d1da810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c52ffd640 0 client.25112 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 --2- 172.21.15.16:0/71003188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f99d1cc660 0x55f99d1ddd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 >> 172.21.15.16:0/71003188 conn(0x55f99d1c7cd0 msgr2=0x55f99d1c80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 shutdown_connections 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.726+0000 7f4c7cdb4740 1 -- 172.21.15.16:0/71003188 wait complete. 2024-04-25T01:29:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7cdb4740 20 client.25112 trim_cache size 0 max 16384 2024-04-25T01:29:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:29:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:29:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:29:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:29:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4c5c0a2c30 msgr2=0x7f4c5c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4c5c0a2c30 0x7f4c5c0a50f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4c74197d60 tx=0x7f4c68005a20 comp rx=0 tx=0).stop 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c741006e0 msgr2=0x7f4c74196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c741006e0 0x7f4c74196850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4c74101730 tx=0x7f4c6400dbb0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 shutdown_connections 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4c5c0a2c30 0x7f4c5c0a50f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c74102290 0x7f4c7419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c741018d0 0x7f4c74196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 --2- 172.21.15.16:0/865870532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c741006e0 0x7f4c74196850 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.727+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 >> 172.21.15.16:0/865870532 conn(0x7f4c740fbc50 msgr2=0x7f4c74104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.728+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 shutdown_connections 2024-04-25T01:29:20.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:20.728+0000 7f4c7a0a5640 1 -- 172.21.15.16:0/865870532 wait complete. 2024-04-25T01:29:21.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:21 smithi038 ceph-mon[32841]: pgmap v417: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 174 MiB/s wr, 95 op/s 2024-04-25T01:29:21.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:21 smithi156 ceph-mon[32631]: pgmap v417: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 174 MiB/s wr, 95 op/s 2024-04-25T01:29:21.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:21 smithi016 ceph-mon[29153]: pgmap v417: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 174 MiB/s wr, 95 op/s 2024-04-25T01:29:23.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:23 smithi038 ceph-mon[32841]: pgmap v418: 97 pgs: 97 active+clean; 7.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 155 MiB/s wr, 86 op/s 2024-04-25T01:29:23.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:23 smithi156 ceph-mon[32631]: pgmap v418: 97 pgs: 97 active+clean; 7.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 155 MiB/s wr, 86 op/s 2024-04-25T01:29:23.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:23 smithi016 ceph-mon[29153]: pgmap v418: 97 pgs: 97 active+clean; 7.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 155 MiB/s wr, 86 op/s 2024-04-25T01:29:25.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:25 smithi038 ceph-mon[32841]: pgmap v419: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 181 MiB/s wr, 98 op/s 2024-04-25T01:29:25.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:25 smithi156 ceph-mon[32631]: pgmap v419: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 181 MiB/s wr, 98 op/s 2024-04-25T01:29:25.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:25 smithi016 ceph-mon[29153]: pgmap v419: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 181 MiB/s wr, 98 op/s 2024-04-25T01:29:27.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:27 smithi038 ceph-mon[32841]: pgmap v420: 97 pgs: 97 active+clean; 7.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 149 MiB/s wr, 84 op/s 2024-04-25T01:29:27.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:27 smithi156 ceph-mon[32631]: pgmap v420: 97 pgs: 97 active+clean; 7.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 149 MiB/s wr, 84 op/s 2024-04-25T01:29:27.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:27 smithi016 ceph-mon[29153]: pgmap v420: 97 pgs: 97 active+clean; 7.8 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 149 MiB/s wr, 84 op/s 2024-04-25T01:29:29.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:29 smithi038 ceph-mon[32841]: pgmap v421: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 174 MiB/s wr, 96 op/s 2024-04-25T01:29:29.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:29 smithi156 ceph-mon[32631]: pgmap v421: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 174 MiB/s wr, 96 op/s 2024-04-25T01:29:29.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:29 smithi016 ceph-mon[29153]: pgmap v421: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 174 MiB/s wr, 96 op/s 2024-04-25T01:29:31.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:31 smithi038 ceph-mon[32841]: pgmap v422: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 175 MiB/s wr, 96 op/s 2024-04-25T01:29:31.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:31 smithi156 ceph-mon[32631]: pgmap v422: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 175 MiB/s wr, 96 op/s 2024-04-25T01:29:31.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:31 smithi016 ceph-mon[29153]: pgmap v422: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 175 MiB/s wr, 96 op/s 2024-04-25T01:29:33.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:33 smithi156 ceph-mon[32631]: pgmap v423: 97 pgs: 97 active+clean; 8.8 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 158 MiB/s wr, 89 op/s 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 -- 172.21.15.16:0/1832597739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 msgr2=0x7f55ec101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 --2- 172.21.15.16:0/1832597739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec101bf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009900 tx=0x7f55e802a410 comp rx=0 tx=0).stop 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 -- 172.21.15.16:0/1832597739 shutdown_connections 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 --2- 172.21.15.16:0/1832597739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec103380 0x7f55ec109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 --2- 172.21.15.16:0/1832597739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec1029c0 0x7f55ec102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 --2- 172.21.15.16:0/1832597739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec101bf0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 -- 172.21.15.16:0/1832597739 >> 172.21.15.16:0/1832597739 conn(0x7f55ec0fcd60 msgr2=0x7f55ec0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 -- 172.21.15.16:0/1832597739 shutdown_connections 2024-04-25T01:29:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.862+0000 7f55f37a5640 1 -- 172.21.15.16:0/1832597739 wait complete. 2024-04-25T01:29:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.863+0000 7f55f37a5640 1 Processor -- start 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.863+0000 7f55f37a5640 1 -- start start 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.863+0000 7f55f37a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec195730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f37a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec1029c0 0x7f55ec195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f37a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec103380 0x7f55ec19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec195730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec195730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47938/0 (socket says 172.21.15.16:47938) 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 -- 172.21.15.16:0/3215539962 learned_addr learned my addr 172.21.15.16:0/3215539962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10b120 con 0x7f55ec1017d0 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10afa0 con 0x7f55ec103380 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10b2a0 con 0x7f55ec1029c0 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f0d40640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec1029c0 0x7f55ec195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 -- 172.21.15.16:0/3215539962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec1029c0 msgr2=0x7f55ec195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec1029c0 0x7f55ec195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 -- 172.21.15.16:0/3215539962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec103380 msgr2=0x7f55ec19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec103380 0x7f55ec19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f1541640 1 -- 172.21.15.16:0/3215539962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec19d350 con 0x7f55ec1017d0 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.864+0000 7f55f0d40640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec1029c0 0x7f55ec195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.865+0000 7f55f1541640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec195730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f55e80314e0 tx=0x7f55e8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.865+0000 7f55e27fc640 1 -- 172.21.15.16:0/3215539962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e803f030 con 0x7f55ec1017d0 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.865+0000 7f55e27fc640 1 -- 172.21.15.16:0/3215539962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55e8003aa0 con 0x7f55ec1017d0 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.865+0000 7f55e27fc640 1 -- 172.21.15.16:0/3215539962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8035070 con 0x7f55ec1017d0 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.865+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ec19d5e0 con 0x7f55ec1017d0 2024-04-25T01:29:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.865+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ec19da60 con 0x7f55ec1017d0 2024-04-25T01:29:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.868+0000 7f55f658e740 1 Processor -- start 2024-04-25T01:29:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.868+0000 7f55f658e740 1 -- start start 2024-04-25T01:29:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55e27fc640 1 -- 172.21.15.16:0/3215539962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f55e8042030 con 0x7f55ec1017d0 2024-04-25T01:29:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f658e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bae270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f658e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f658e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40bb5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f658e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559d40b11cb0 con 0x559d40bb39b0 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f658e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559d40aba5b0 con 0x559d40bade50 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f658e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559d40b0dc70 con 0x559d40bae840 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:49150/0 (socket says 172.21.15.16:49150) 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 -- 172.21.15.16:0/2021297670 learned_addr learned my addr 172.21.15.16:0/2021297670 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f0d40640 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40bb5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1541640 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 -- 172.21.15.16:0/2021297670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 msgr2=0x559d40bb3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 -- 172.21.15.16:0/2021297670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 msgr2=0x559d40bb5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40bb5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55f1d42640 1 -- 172.21.15.16:0/2021297670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d40826210 con 0x559d40bade50 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.869+0000 7f55e27fc640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55d80a2cb0 0x7f55d80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.870+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bae270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009ea0 tx=0x7f55e4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9a02bb59eae96cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.870+0000 7f55f1541640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55d80a2cb0 0x7f55d80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.870+0000 7f55e27fc640 1 -- 172.21.15.16:0/3215539962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55e80ef830 con 0x7f55ec1017d0 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.870+0000 7f55d77fe640 1 -- 172.21.15.16:0/2021297670 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4004d20 con 0x559d40bade50 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.870+0000 7f55d77fe640 1 -- 172.21.15.16:0/2021297670 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55e400ee60 con 0x559d40bade50 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.870+0000 7f55d77fe640 1 -- 172.21.15.16:0/2021297670 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4010830 con 0x559d40bade50 2024-04-25T01:29:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 -- 172.21.15.16:0/2021297670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 msgr2=0x559d40bae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bae270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009ea0 tx=0x7f55e4009f90 comp rx=0 tx=0).stop 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f1541640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55d80a2cb0 0x7f55d80a5170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55e80ed680 tx=0x7f55e80dc040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 -- 172.21.15.16:0/2021297670 shutdown_connections 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40bb5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 --2- 172.21.15.16:0/2021297670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bae270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 -- 172.21.15.16:0/2021297670 >> 172.21.15.16:0/2021297670 conn(0x559d40ba94c0 msgr2=0x559d40ba98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 -- 172.21.15.16:0/2021297670 shutdown_connections 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 -- 172.21.15.16:0/2021297670 wait complete. 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 -- start start 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.871+0000 7f55f658e740 1 Processor -- start 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 0x559d40bb8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f1d42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 0x559d40bb8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f1d42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 0x559d40bb8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47960/0 (socket says 172.21.15.16:47960) 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f1d42640 1 -- 172.21.15.16:0/1939570073 learned_addr learned my addr 172.21.15.16:0/1939570073 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f0d40640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559d40bb6960 con 0x559d40bade50 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559d40acdb50 con 0x559d40bb39b0 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559d40bb6ae0 con 0x559d40bae840 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f658e740 10 client.0 _ll_register_callbacks cb 0x559d40b08ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.872+0000 7f55f1d42640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f1d42640 1 -- 172.21.15.16:0/1939570073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 msgr2=0x559d40bb8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f1d42640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f1d42640 1 -- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 msgr2=0x559d40bbf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f1d42640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f1d42640 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d408f2620 con 0x559d40bade50 2024-04-25T01:29:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e400ee60 con 0x559d40bade50 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f1d42640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 0x559d40bb8250 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f55e4005460 tx=0x7f55e4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a4c4f67e6b688e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55d4ff9640 10 client.15453 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55d4ff9640 1 -- 172.21.15.16:0/1939570073 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e402c060 con 0x559d40bade50 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55d4ff9640 1 -- 172.21.15.16:0/1939570073 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55e4010530 con 0x559d40bade50 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55d4ff9640 1 -- 172.21.15.16:0/1939570073 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f55e401d040 con 0x559d40bade50 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f658e740 10 client.15453 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.873+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=22}) -- 0x559d40bc2ae0 con 0x559d40bade50 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55d4ff9640 1 -- 172.21.15.16:0/1939570073 <== mon.0 v2:172.21.15.16:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f55e4010730 con 0x559d40bade50 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f658e740 10 client.15453 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f658e740 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f658e740 20 client.15453 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40bc3e30 0x559d40bc6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f658e740 4 client.15453 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d40b1f1f0 con 0x559d40bc3e30 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f0d40640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40bc3e30 0x559d40bc6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55d4ff9640 10 client.15453 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.874+0000 7f55f0d40640 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40bc3e30 0x559d40bc6220 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70c05fbd889b612d server_cookie=89cabea9142df0bc in_seq=0 out_seq=0 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55d4ff9640 1 -- 172.21.15.16:0/1939570073 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d40b1f1f0 con 0x559d40bc3e30 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55d4ff9640 10 client.15453 handle_command_reply: tid=0 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 client.15453 shutdown 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 msgr2=0x559d40bb8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 0x559d40bb8250 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f55e4005460 tx=0x7f55e4009b80 comp rx=0 tx=0).stop 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 shutdown_connections 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40bc3e30 0x559d40bc6220 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bb8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 --2- 172.21.15.16:0/1939570073 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bade50 0x559d40bb8250 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 >> 172.21.15.16:0/1939570073 conn(0x559d40ba94c0 msgr2=0x559d40ba85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.875+0000 7f55d4ff9640 0 client.15453 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.876+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 shutdown_connections 2024-04-25T01:29:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.876+0000 7f55f658e740 1 -- 172.21.15.16:0/1939570073 wait complete. 2024-04-25T01:29:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.876+0000 7f55f658e740 20 client.15453 trim_cache size 0 max 16384 2024-04-25T01:29:33.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:33 smithi016 ceph-mon[29153]: pgmap v423: 97 pgs: 97 active+clean; 8.8 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 158 MiB/s wr, 89 op/s 2024-04-25T01:29:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 Processor -- start 2024-04-25T01:29:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 -- start start 2024-04-25T01:29:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bb86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bbf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40ba8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1a91e0 con 0x559d40bb39b0 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1a6ab0 con 0x559d40bade50 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f658e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1a4900 con 0x559d40bae840 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f1d42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40ba8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f1d42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40ba8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47968/0 (socket says 172.21.15.16:47968) 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f1d42640 1 -- 172.21.15.16:0/3473233918 learned_addr learned my addr 172.21.15.16:0/3473233918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f0d40640 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bbf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.900+0000 7f55f1d42640 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bb86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f1d42640 1 -- 172.21.15.16:0/3473233918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 msgr2=0x559d40bbf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f1d42640 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bbf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f1d42640 1 -- 172.21.15.16:0/3473233918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 msgr2=0x559d40bb86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f1d42640 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bb86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f1d42640 1 -- 172.21.15.16:0/3473233918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d40ba9060 con 0x559d40bb39b0 2024-04-25T01:29:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f0d40640 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bbf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55f1d42640 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40ba8440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f55e4005430 tx=0x7f55e4025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2215ec08921f3a38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55d6ffd640 1 -- 172.21.15.16:0/3473233918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4010c80 con 0x559d40bb39b0 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55d6ffd640 1 -- 172.21.15.16:0/3473233918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55e400ee60 con 0x559d40bb39b0 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.901+0000 7f55d6ffd640 1 -- 172.21.15.16:0/3473233918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4018850 con 0x559d40bb39b0 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 -- 172.21.15.16:0/3473233918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 msgr2=0x559d40ba8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40ba8440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f55e4005430 tx=0x7f55e4025000 comp rx=0 tx=0).stop 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 -- 172.21.15.16:0/3473233918 shutdown_connections 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bb39b0 0x559d40ba8440 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bae840 0x559d40bbf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 --2- 172.21.15.16:0/3473233918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bade50 0x559d40bb86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 -- 172.21.15.16:0/3473233918 >> 172.21.15.16:0/3473233918 conn(0x559d40ba94c0 msgr2=0x559d40ba7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 -- 172.21.15.16:0/3473233918 shutdown_connections 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.902+0000 7f55f658e740 1 -- 172.21.15.16:0/3473233918 wait complete. 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 -- start start 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 Processor -- start 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bade50 0x559d40bbcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 0x559d40bbd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbe140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1a4f70 con 0x559d40bae840 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1a63e0 con 0x559d40bb39b0 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1a5640 con 0x559d40bade50 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f658e740 10 client.0 _ll_register_callbacks cb 0x7f55dc00d110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f0d40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 0x559d40bbd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f0d40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 0x559d40bbd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47976/0 (socket says 172.21.15.16:47976) 2024-04-25T01:29:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f0d40640 1 -- 172.21.15.16:0/2117199486 learned_addr learned my addr 172.21.15.16:0/2117199486 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.903+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bade50 0x559d40bbcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbe140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 -- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 msgr2=0x559d40bbe140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbe140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 -- 172.21.15.16:0/2117199486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 msgr2=0x559d40bbd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 0x559d40bbd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 -- 172.21.15.16:0/2117199486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d40bbe730 con 0x559d40bade50 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f0d40640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 0x559d40bbd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f1d42640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bade50 0x559d40bbcbf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f55e400a010 tx=0x7f55e4009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18ba33876e3196af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55d4ff9640 1 -- 172.21.15.16:0/2117199486 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4010c80 con 0x559d40bade50 2024-04-25T01:29:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55d4ff9640 1 -- 172.21.15.16:0/2117199486 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55e4010430 con 0x559d40bade50 2024-04-25T01:29:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.904+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e400ee60 con 0x559d40bade50 2024-04-25T01:29:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.905+0000 7f55d4ff9640 1 -- 172.21.15.16:0/2117199486 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f55e402c030 con 0x559d40bade50 2024-04-25T01:29:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.905+0000 7f55f658e740 10 client.25099 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.905+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x559d40bbfa40 con 0x559d40bade50 2024-04-25T01:29:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.905+0000 7f55d4ff9640 1 -- 172.21.15.16:0/2117199486 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55e400ee60 con 0x559d40bade50 2024-04-25T01:29:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55d4ff9640 1 -- 172.21.15.16:0/2117199486 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f55e40318e0 con 0x559d40bade50 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f658e740 10 client.25099 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f658e740 10 client.25099 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f658e740 20 client.25099 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f658e740 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40c2a870 0x559d40c2cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f658e740 4 client.25099 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d40b19540 con 0x559d40c2a870 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.906+0000 7f55f0d40640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40c2a870 0x559d40c2cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.907+0000 7f55d4ff9640 10 client.25099 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.907+0000 7f55f0d40640 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40c2a870 0x559d40c2cc90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c4c66ac86b2665 server_cookie=aa64cf23d7260861 in_seq=0 out_seq=0 2024-04-25T01:29:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55d4ff9640 1 -- 172.21.15.16:0/2117199486 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d40b19540 con 0x559d40c2a870 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55d4ff9640 10 client.25099 handle_command_reply: tid=0 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 client.25099 shutdown 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bade50 msgr2=0x559d40bbcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bade50 0x559d40bbcbf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f55e400a010 tx=0x7f55e4009610 comp rx=0 tx=0).stop 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 shutdown_connections 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559d40c2a870 0x559d40c2cc90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d40bb39b0 0x559d40bbe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d40bae840 0x559d40bbd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 --2- 172.21.15.16:0/2117199486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d40bade50 0x559d40bbcbf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55d4ff9640 0 client.25099 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 >> 172.21.15.16:0/2117199486 conn(0x559d40ba94c0 msgr2=0x559d40ba85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 shutdown_connections 2024-04-25T01:29:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 1 -- 172.21.15.16:0/2117199486 wait complete. 2024-04-25T01:29:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.915+0000 7f55f658e740 20 client.25099 trim_cache size 0 max 16384 2024-04-25T01:29:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:29:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:29:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:29:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55d80a2cb0 msgr2=0x7f55d80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55d80a2cb0 0x7f55d80a5170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55e80ed680 tx=0x7f55e80dc040 comp rx=0 tx=0).stop 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 msgr2=0x7f55ec195730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec195730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f55e80314e0 tx=0x7f55e8004270 comp rx=0 tx=0).stop 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 shutdown_connections 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55d80a2cb0 0x7f55d80a5170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec103380 0x7f55ec19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55ec1029c0 0x7f55ec195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 --2- 172.21.15.16:0/3215539962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55ec1017d0 0x7f55ec195730 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 >> 172.21.15.16:0/3215539962 conn(0x7f55ec0fcd60 msgr2=0x7f55ec105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 shutdown_connections 2024-04-25T01:29:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:33.916+0000 7f55f37a5640 1 -- 172.21.15.16:0/3215539962 wait complete. 2024-04-25T01:29:33.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:33 smithi038 ceph-mon[32841]: pgmap v423: 97 pgs: 97 active+clean; 8.8 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 158 MiB/s wr, 89 op/s 2024-04-25T01:29:35.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:35 smithi156 ceph-mon[32631]: pgmap v424: 97 pgs: 97 active+clean; 9.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 189 MiB/s wr, 102 op/s 2024-04-25T01:29:35.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:35 smithi016 ceph-mon[29153]: pgmap v424: 97 pgs: 97 active+clean; 9.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 189 MiB/s wr, 102 op/s 2024-04-25T01:29:35.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:35 smithi038 ceph-mon[32841]: pgmap v424: 97 pgs: 97 active+clean; 9.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 189 MiB/s wr, 102 op/s 2024-04-25T01:29:37.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:37 smithi156 ceph-mon[32631]: pgmap v425: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 151 MiB/s wr, 84 op/s 2024-04-25T01:29:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:37 smithi016 ceph-mon[29153]: pgmap v425: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 151 MiB/s wr, 84 op/s 2024-04-25T01:29:37.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:37 smithi038 ceph-mon[32841]: pgmap v425: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 151 MiB/s wr, 84 op/s 2024-04-25T01:29:39.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:39 smithi156 ceph-mon[32631]: pgmap v426: 97 pgs: 97 active+clean; 9.8 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 MiB/s wr, 137 op/s 2024-04-25T01:29:39.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:39 smithi016 ceph-mon[29153]: pgmap v426: 97 pgs: 97 active+clean; 9.8 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 MiB/s wr, 137 op/s 2024-04-25T01:29:39.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:39 smithi038 ceph-mon[32841]: pgmap v426: 97 pgs: 97 active+clean; 9.8 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 MiB/s wr, 137 op/s 2024-04-25T01:29:41.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:41 smithi156 ceph-mon[32631]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 160 MiB/s wr, 246 op/s 2024-04-25T01:29:41.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:41 smithi016 ceph-mon[29153]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 160 MiB/s wr, 246 op/s 2024-04-25T01:29:41.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:41 smithi038 ceph-mon[32841]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 160 MiB/s wr, 246 op/s 2024-04-25T01:29:42.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:42 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:42.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:42 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:42.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:42 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:43.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:43 smithi156 ceph-mon[32631]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 135 MiB/s wr, 376 op/s 2024-04-25T01:29:43.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:43.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:43.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:29:43.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:43 smithi016 ceph-mon[29153]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 135 MiB/s wr, 376 op/s 2024-04-25T01:29:43.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:43.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:43.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:29:43.865 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:43 smithi038 ceph-mon[32841]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 135 MiB/s wr, 376 op/s 2024-04-25T01:29:43.865 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:43.865 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:43.865 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:29:45.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:45 smithi156 ceph-mon[32631]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 140 MiB/s wr, 709 op/s 2024-04-25T01:29:45.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:45 smithi016 ceph-mon[29153]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 140 MiB/s wr, 709 op/s 2024-04-25T01:29:45.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:45 smithi038 ceph-mon[32841]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 140 MiB/s wr, 709 op/s 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 -- 172.21.15.16:0/2481009156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 msgr2=0x7fbf6c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 --2- 172.21.15.16:0/2481009156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 0x7fbf6c101a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009900 tx=0x7fbf5c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 -- 172.21.15.16:0/2481009156 shutdown_connections 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 --2- 172.21.15.16:0/2481009156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 0x7fbf6c109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 --2- 172.21.15.16:0/2481009156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 --2- 172.21.15.16:0/2481009156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 0x7fbf6c101a20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.060+0000 7fbf716ee640 1 -- 172.21.15.16:0/2481009156 >> 172.21.15.16:0/2481009156 conn(0x7fbf6c0fcb70 msgr2=0x7fbf6c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.061+0000 7fbf716ee640 1 -- 172.21.15.16:0/2481009156 shutdown_connections 2024-04-25T01:29:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.061+0000 7fbf716ee640 1 -- 172.21.15.16:0/2481009156 wait complete. 2024-04-25T01:29:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.061+0000 7fbf716ee640 1 Processor -- start 2024-04-25T01:29:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.062+0000 7fbf716ee640 1 -- start start 2024-04-25T01:29:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.062+0000 7fbf716ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 0x7fbf6c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.062+0000 7fbf716ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.062+0000 7fbf716ee640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 0x7fbf6c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf6a7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf6a7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40584/0 (socket says 172.21.15.16:40584) 2024-04-25T01:29:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf6affd640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 0x7fbf6c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf6a7fc640 1 -- 172.21.15.16:0/3088330319 learned_addr learned my addr 172.21.15.16:0/3088330319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 0x7fbf6c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c10d170 con 0x7fbf6c101600 2024-04-25T01:29:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c10cff0 con 0x7fbf6c1031b0 2024-04-25T01:29:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.063+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c10d2f0 con 0x7fbf6c1027f0 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6b7fe640 1 -- 172.21.15.16:0/3088330319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 msgr2=0x7fbf6c197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6b7fe640 1 -- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 msgr2=0x7fbf6c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 0x7fbf6c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6b7fe640 1 -- 172.21.15.16:0/3088330319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c19f2b0 con 0x7fbf6c1031b0 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 0x7fbf6c19ec40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009fd0 tx=0x7fbf6400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf53fff640 1 -- 172.21.15.16:0/3088330319 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400ce20 con 0x7fbf6c1031b0 2024-04-25T01:29:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf53fff640 1 -- 172.21.15.16:0/3088330319 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf64005020 con 0x7fbf6c1031b0 2024-04-25T01:29:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6c0739d0 con 0x7fbf6c1031b0 2024-04-25T01:29:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.064+0000 7fbf53fff640 1 -- 172.21.15.16:0/3088330319 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64010690 con 0x7fbf6c1031b0 2024-04-25T01:29:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.065+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf6c073f20 con 0x7fbf6c1031b0 2024-04-25T01:29:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.068+0000 7fbf53fff640 1 -- 172.21.15.16:0/3088330319 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbf640051d0 con 0x7fbf6c1031b0 2024-04-25T01:29:47.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.069+0000 7fbf53fff640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf540a2cc0 0x7fbf540a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.069+0000 7fbf53fff640 1 -- 172.21.15.16:0/3088330319 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbf640cf700 con 0x7fbf6c1031b0 2024-04-25T01:29:47.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.069+0000 7fbf6affd640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf540a2cc0 0x7fbf540a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.048 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.069+0000 7fbf743f7740 1 Processor -- start 2024-04-25T01:29:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 -- start start 2024-04-25T01:29:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 0x565488452450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf540a2cc0 0x7fbf540a5180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c005ab0 tx=0x7fbf5c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x565488457520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x56548845a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5654883b5cb0 con 0x565488452030 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56548835e5b0 con 0x565488452a20 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf743f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5654883b1c70 con 0x565488457b90 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 0x565488452450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 0x565488452450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43366/0 (socket says 172.21.15.16:43366) 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 -- 172.21.15.16:0/2195226720 learned_addr learned my addr 172.21.15.16:0/2195226720 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x565488457520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x56548845a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 -- 172.21.15.16:0/2195226720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 msgr2=0x56548845a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x56548845a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.070+0000 7fbf6affd640 1 -- 172.21.15.16:0/2195226720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 msgr2=0x565488457520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf6affd640 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x565488457520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf6affd640 1 -- 172.21.15.16:0/2195226720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654880ca210 con 0x565488452030 2024-04-25T01:29:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x565488457520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf6affd640 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 0x565488452450 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c102650 tx=0x7fbf5c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9aa5cd318d7c84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf517fa640 1 -- 172.21.15.16:0/2195226720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c035040 con 0x565488452030 2024-04-25T01:29:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf517fa640 1 -- 172.21.15.16:0/2195226720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf5c04e070 con 0x565488452030 2024-04-25T01:29:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.071+0000 7fbf517fa640 1 -- 172.21.15.16:0/2195226720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c003e20 con 0x565488452030 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.072+0000 7fbf743f7740 1 -- 172.21.15.16:0/2195226720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 msgr2=0x565488452450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.072+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 0x565488452450 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c102650 tx=0x7fbf5c03d000 comp rx=0 tx=0).stop 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.072+0000 7fbf743f7740 1 -- 172.21.15.16:0/2195226720 shutdown_connections 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.072+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x56548845a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.072+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x565488457520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.072+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2195226720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452030 0x565488452450 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.073+0000 7fbf743f7740 1 -- 172.21.15.16:0/2195226720 >> 172.21.15.16:0/2195226720 conn(0x56548844d6a0 msgr2=0x56548844daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.073+0000 7fbf743f7740 1 -- 172.21.15.16:0/2195226720 shutdown_connections 2024-04-25T01:29:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.073+0000 7fbf743f7740 1 -- 172.21.15.16:0/2195226720 wait complete. 2024-04-25T01:29:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.074+0000 7fbf743f7740 1 -- start start 2024-04-25T01:29:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.074+0000 7fbf743f7740 1 Processor -- start 2024-04-25T01:29:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.074+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x56548845c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x56548845c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488457b90 0x5654884639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf743f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56548845aae0 con 0x565488457b90 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf743f7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x565488371b50 con 0x565488452a20 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf743f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56548845ac60 con 0x565488452030 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x56548845c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x56548845c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x56548845c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40600/0 (socket says 172.21.15.16:40600) 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 -- 172.21.15.16:0/1171528251 learned_addr learned my addr 172.21.15.16:0/1171528251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488457b90 0x5654884639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf743f7740 10 client.0 _ll_register_callbacks cb 0x5654883acca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 -- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 msgr2=0x56548845c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x56548845c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 -- 172.21.15.16:0/1171528251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488457b90 msgr2=0x5654884639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488457b90 0x5654884639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.075+0000 7fbf6affd640 1 -- 172.21.15.16:0/1171528251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565488196620 con 0x565488452030 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf6affd640 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x56548845c420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c004650 tx=0x7fbf5c0035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb60149954534e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf4eff5640 1 -- 172.21.15.16:0/1171528251 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c035070 con 0x565488452030 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf4eff5640 1 -- 172.21.15.16:0/1171528251 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbf5c004720 con 0x565488452030 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf5c04e070 con 0x565488452030 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf4eff5640 1 -- 172.21.15.16:0/1171528251 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fbf5c003e60 con 0x565488452030 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf743f7740 10 client.25111 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.076+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56548845aae0 con 0x565488452030 2024-04-25T01:29:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf4eff5640 1 -- 172.21.15.16:0/1171528251 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf5c004dd0 con 0x565488452030 2024-04-25T01:29:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf4eff5640 1 -- 172.21.15.16:0/1171528251 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fbf5c032c00 con 0x565488452030 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf743f7740 10 client.25111 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf743f7740 10 client.25111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf743f7740 20 client.25111 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf743f7740 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x565488467f10 0x56548846a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf743f7740 4 client.25111 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbf5c04e070 con 0x565488467f10 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.077+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x565488467f10 0x56548846a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.078+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x565488467f10 0x56548846a300 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e0fb40959aca063 server_cookie=fa967f0372c05e in_seq=0 out_seq=0 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.078+0000 7fbf4eff5640 10 client.25111 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf4eff5640 1 -- 172.21.15.16:0/1171528251 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbf5c04e070 con 0x565488467f10 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf4eff5640 10 client.25111 handle_command_reply: tid=0 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 client.25111 shutdown 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 msgr2=0x56548845c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x56548845c420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c004650 tx=0x7fbf5c0035f0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 shutdown_connections 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488457b90 0x5654884639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x565488467f10 0x56548846a300 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf4eff5640 0 client.25111 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452a20 0x56548845c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 --2- 172.21.15.16:0/1171528251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x56548845c420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 >> 172.21.15.16:0/1171528251 conn(0x56548844d6a0 msgr2=0x56548844c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 shutdown_connections 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.079+0000 7fbf743f7740 1 -- 172.21.15.16:0/1171528251 wait complete. 2024-04-25T01:29:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.080+0000 7fbf743f7740 20 client.25111 trim_cache size 0 max 16384 2024-04-25T01:29:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.098+0000 7fbf743f7740 1 Processor -- start 2024-04-25T01:29:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.098+0000 7fbf743f7740 1 -- start start 2024-04-25T01:29:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452030 0x56548845c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488450650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x565488450b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452030 0x56548845c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf743f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c1a8f20 con 0x565488452a20 2024-04-25T01:29:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf743f7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c1a67f0 con 0x565488452030 2024-04-25T01:29:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf743f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c1a4640 con 0x565488457b90 2024-04-25T01:29:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452030 0x56548845c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:56386/0 (socket says 172.21.15.16:56386) 2024-04-25T01:29:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488450650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x565488450b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6affd640 1 -- 172.21.15.16:0/2102309713 learned_addr learned my addr 172.21.15.16:0/2102309713 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6b7fe640 1 -- 172.21.15.16:0/2102309713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452030 msgr2=0x56548845c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452030 0x56548845c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6b7fe640 1 -- 172.21.15.16:0/2102309713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 msgr2=0x565488450650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488450650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6b7fe640 1 -- 172.21.15.16:0/2102309713 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565488453280 con 0x565488457b90 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.099+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488450650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.100+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x565488450b90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400c760 tx=0x7fbf640cf040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29692dfe4af0f332 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.100+0000 7fbf50ff9640 1 -- 172.21.15.16:0/2102309713 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6409a1e0 con 0x565488457b90 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.100+0000 7fbf50ff9640 1 -- 172.21.15.16:0/2102309713 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf640b7020 con 0x565488457b90 2024-04-25T01:29:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.100+0000 7fbf50ff9640 1 -- 172.21.15.16:0/2102309713 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf640850b0 con 0x565488457b90 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 -- 172.21.15.16:0/2102309713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 msgr2=0x565488450b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x565488450b90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400c760 tx=0x7fbf640cf040 comp rx=0 tx=0).stop 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 -- 172.21.15.16:0/2102309713 shutdown_connections 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488457b90 0x565488450b90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488450650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 --2- 172.21.15.16:0/2102309713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488452030 0x56548845c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 -- 172.21.15.16:0/2102309713 >> 172.21.15.16:0/2102309713 conn(0x56548844d6a0 msgr2=0x56548845c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 -- 172.21.15.16:0/2102309713 shutdown_connections 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 -- 172.21.15.16:0/2102309713 wait complete. 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 -- start start 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 Processor -- start 2024-04-25T01:29:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x5654884607d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.101+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488460d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf743f7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488457b90 0x565488461d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf743f7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c1a4cb0 con 0x565488452a20 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x5654884607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x5654884607d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40620/0 (socket says 172.21.15.16:40620) 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6affd640 1 -- 172.21.15.16:0/3294029876 learned_addr learned my addr 172.21.15.16:0/3294029876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c1a6120 con 0x565488457b90 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488460d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c1a5380 con 0x565488452030 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf743f7740 10 client.0 _ll_register_callbacks cb 0x7fbf64098180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488457b90 0x565488461d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 -- 172.21.15.16:0/3294029876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 msgr2=0x5654884607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x5654884607d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 -- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488457b90 msgr2=0x565488461d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488457b90 0x565488461d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 -- 172.21.15.16:0/3294029876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565488462310 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6b7fe640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488457b90 0x565488461d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf6a7fc640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488460d10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x56548845d900 tx=0x7fbf600044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7777ee12eb280d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf4eff5640 1 -- 172.21.15.16:0/3294029876 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf60005020 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf4eff5640 1 -- 172.21.15.16:0/3294029876 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbf600051c0 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.102+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf640b7020 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf4eff5640 1 -- 172.21.15.16:0/3294029876 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fbf6002d070 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 10 client.15477 fetch_fsmap learned FSMap version 22 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5654884ce730 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf4eff5640 1 -- 172.21.15.16:0/3294029876 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf640b7020 con 0x565488452a20 2024-04-25T01:29:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf4eff5640 1 -- 172.21.15.16:0/3294029876 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fbf60003650 con 0x565488452a20 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 10 client.15477 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 10 client.15477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 20 client.15477 populate_metadata read hostname 'smithi016' 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5654884cf150 0x5654884d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 4 client.15477 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.103+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654883c31f0 con 0x5654884cf150 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.104+0000 7fbf6affd640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5654884cf150 0x5654884d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.104+0000 7fbf6affd640 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5654884cf150 0x5654884d1570 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b75b24cea6bf46 server_cookie=8072fe7f797add02 in_seq=0 out_seq=0 2024-04-25T01:29:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.104+0000 7fbf4eff5640 10 client.15477 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.112+0000 7fbf4eff5640 1 -- 172.21.15.16:0/3294029876 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654883c31f0 con 0x5654884cf150 2024-04-25T01:29:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.112+0000 7fbf4eff5640 10 client.15477 handle_command_reply: tid=0 2024-04-25T01:29:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.112+0000 7fbf743f7740 1 client.15477 shutdown 2024-04-25T01:29:47.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 msgr2=0x565488460d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488460d10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x56548845d900 tx=0x7fbf600044d0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 shutdown_connections 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5654884cf150 0x5654884d1570 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565488457b90 0x565488461d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565488452a20 0x565488460d10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 --2- 172.21.15.16:0/3294029876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565488452030 0x5654884607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 >> 172.21.15.16:0/3294029876 conn(0x56548844d6a0 msgr2=0x56548845bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf4eff5640 0 client.15477 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 shutdown_connections 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 1 -- 172.21.15.16:0/3294029876 wait complete. 2024-04-25T01:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.113+0000 7fbf743f7740 20 client.15477 trim_cache size 0 max 16384 2024-04-25T01:29:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:29:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:29:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:29:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:29:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf540a2cc0 msgr2=0x7fbf540a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf540a2cc0 0x7fbf540a5180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c005ab0 tx=0x7fbf5c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 msgr2=0x7fbf6c19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 0x7fbf6c19ec40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009fd0 tx=0x7fbf6400ef10 comp rx=0 tx=0).stop 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 shutdown_connections 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf540a2cc0 0x7fbf540a5180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf6c1031b0 0x7fbf6c19ec40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c1027f0 0x7fbf6c197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 --2- 172.21.15.16:0/3088330319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c101600 0x7fbf6c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 >> 172.21.15.16:0/3088330319 conn(0x7fbf6c0fcb70 msgr2=0x7fbf6c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.114+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 shutdown_connections 2024-04-25T01:29:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:29:47.115+0000 7fbf716ee640 1 -- 172.21.15.16:0/3088330319 wait complete. 2024-04-25T01:29:47.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:47 smithi156 ceph-mon[32631]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 100 MiB/s wr, 752 op/s 2024-04-25T01:29:47.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:47 smithi016 ceph-mon[29153]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 100 MiB/s wr, 752 op/s 2024-04-25T01:29:47.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:47 smithi038 ceph-mon[32841]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 100 MiB/s wr, 752 op/s 2024-04-25T01:29:49.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:49 smithi156 ceph-mon[32631]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 104 MiB/s wr, 994 op/s 2024-04-25T01:29:49.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:49 smithi016 ceph-mon[29153]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 104 MiB/s wr, 994 op/s 2024-04-25T01:29:49.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:49 smithi038 ceph-mon[32841]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 104 MiB/s wr, 994 op/s 2024-04-25T01:29:51.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:51 smithi156 ceph-mon[32631]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 90 MiB/s wr, 1.18k op/s 2024-04-25T01:29:51.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:51 smithi016 ceph-mon[29153]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 90 MiB/s wr, 1.18k op/s 2024-04-25T01:29:51.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:51 smithi038 ceph-mon[32841]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 90 MiB/s wr, 1.18k op/s 2024-04-25T01:29:53.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:53 smithi156 ceph-mon[32631]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-04-25T01:29:53.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:53 smithi016 ceph-mon[29153]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-04-25T01:29:53.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:53 smithi038 ceph-mon[32841]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-04-25T01:29:55.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:55 smithi156 ceph-mon[32631]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-25T01:29:55.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:55 smithi016 ceph-mon[29153]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-25T01:29:55.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:55 smithi038 ceph-mon[32841]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-25T01:29:57.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:57 smithi156 ceph-mon[32631]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:29:57.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:57 smithi016 ceph-mon[29153]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:29:57.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:57 smithi038 ceph-mon[32841]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:29:59.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:29:59 smithi016 ceph-mon[29153]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-25T01:29:59.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:29:59 smithi038 ceph-mon[32841]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-25T01:29:59.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:29:59 smithi156 ceph-mon[32631]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-25T01:30:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 -- 172.21.15.16:0/2865335595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281017d0 msgr2=0x7f2928101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2865335595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281017d0 0x7f2928101bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2918009900 tx=0x7f291802a420 comp rx=0 tx=0).stop 2024-04-25T01:30:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 -- 172.21.15.16:0/2865335595 shutdown_connections 2024-04-25T01:30:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2865335595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 0x7f2928109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2865335595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281029c0 0x7f2928102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.240 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2865335595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281017d0 0x7f2928101bf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.240 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.260+0000 7f292d3d5640 1 -- 172.21.15.16:0/2865335595 >> 172.21.15.16:0/2865335595 conn(0x7f29280fcd60 msgr2=0x7f29280ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:00.240 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.261+0000 7f292d3d5640 1 -- 172.21.15.16:0/2865335595 shutdown_connections 2024-04-25T01:30:00.240 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.261+0000 7f292d3d5640 1 -- 172.21.15.16:0/2865335595 wait complete. 2024-04-25T01:30:00.240 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.261+0000 7f292d3d5640 1 Processor -- start 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.261+0000 7f292d3d5640 1 -- start start 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.261+0000 7f292d3d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 0x7f292819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f292d3d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281029c0 0x7f292819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f292d3d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 0x7f29281a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f292d3d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f292810f5e0 con 0x7f2928103380 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f292d3d5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f292810f460 con 0x7f29281017d0 2024-04-25T01:30:00.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f292d3d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f292810f760 con 0x7f29281029c0 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 0x7f292819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f291ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281029c0 0x7f292819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 0x7f292819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:40192/0 (socket says 172.21.15.16:40192) 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 -- 172.21.15.16:0/2030792818 learned_addr learned my addr 172.21.15.16:0/2030792818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f29277fe640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 0x7f29281a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 -- 172.21.15.16:0/2030792818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281029c0 msgr2=0x7f292819e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281029c0 0x7f292819e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 -- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 msgr2=0x7f29281a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 0x7f29281a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f2926ffd640 1 -- 172.21.15.16:0/2030792818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2928194df0 con 0x7f29281017d0 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f291ffff640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281029c0 0x7f292819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.262+0000 7f29277fe640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 0x7f29281a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.263+0000 7f2926ffd640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 0x7f292819dff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f29180314e0 tx=0x7f2918003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.263+0000 7f2924ff9640 1 -- 172.21.15.16:0/2030792818 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291803f070 con 0x7f29281017d0 2024-04-25T01:30:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.263+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2928195080 con 0x7f29281017d0 2024-04-25T01:30:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.263+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2928195640 con 0x7f29281017d0 2024-04-25T01:30:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.263+0000 7f2924ff9640 1 -- 172.21.15.16:0/2030792818 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2918031e40 con 0x7f29281017d0 2024-04-25T01:30:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.263+0000 7f2924ff9640 1 -- 172.21.15.16:0/2030792818 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918035030 con 0x7f29281017d0 2024-04-25T01:30:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.266+0000 7f2924ff9640 1 -- 172.21.15.16:0/2030792818 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2918042070 con 0x7f29281017d0 2024-04-25T01:30:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.266+0000 7f29301c4740 1 Processor -- start 2024-04-25T01:30:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.266+0000 7f29301c4740 1 -- start start 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29301c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10988d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29301c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109892340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29301c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109894e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29301c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1097f0cb0 con 0x55c10988ce50 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29301c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1097995b0 con 0x55c1098929b0 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29301c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c1097ecc70 con 0x55c10988d840 2024-04-25T01:30:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f2926ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109892340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10988d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f291ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109894e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10988d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38756/0 (socket says 172.21.15.16:38756) 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f2926ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109892340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40438/0 (socket says 172.21.15.16:40438) 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f291ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109894e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:40206/0 (socket says 172.21.15.16:40206) 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f29277fe640 1 -- 172.21.15.16:0/2602313238 learned_addr learned my addr 172.21.15.16:0/2602313238 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f291ffff640 1 -- 172.21.15.16:0/2602313238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 msgr2=0x55c109892340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f291ffff640 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109892340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f2924ff9640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29140a2d30 0x7f29140a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f291ffff640 1 -- 172.21.15.16:0/2602313238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 msgr2=0x55c10988d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.267+0000 7f291ffff640 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10988d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f291ffff640 1 -- 172.21.15.16:0/2602313238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c109505210 con 0x55c1098929b0 2024-04-25T01:30:00.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f29277fe640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29140a2d30 0x7f29140a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f291ffff640 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109894e40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2910006fd0 tx=0x7f291000e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e8334bc95eb6d97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f291d7fa640 1 -- 172.21.15.16:0/2602313238 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910017070 con 0x55c1098929b0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f291d7fa640 1 -- 172.21.15.16:0/2602313238 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2910013920 con 0x55c1098929b0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f291d7fa640 1 -- 172.21.15.16:0/2602313238 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910012770 con 0x55c1098929b0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.268+0000 7f2924ff9640 1 -- 172.21.15.16:0/2030792818 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f29180ef900 con 0x7f29281017d0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 -- 172.21.15.16:0/2602313238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 msgr2=0x55c109894e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29277fe640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29140a2d30 0x7f29140a51f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55c10988e0a0 tx=0x7f2920004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109894e40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2910006fd0 tx=0x7f291000e5a0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 -- 172.21.15.16:0/2602313238 shutdown_connections 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109894e40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109892340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 --2- 172.21.15.16:0/2602313238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10988d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 -- 172.21.15.16:0/2602313238 >> 172.21.15.16:0/2602313238 conn(0x55c1098884c0 msgr2=0x55c1098888c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 -- 172.21.15.16:0/2602313238 shutdown_connections 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.269+0000 7f29301c4740 1 -- 172.21.15.16:0/2602313238 wait complete. 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 -- start start 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 Processor -- start 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c10988ce50 0x55c1098972d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109897810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1098929b0 0x55c10989e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c109895960 con 0x55c1098929b0 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1097acb50 con 0x55c10988ce50 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c109895ae0 con 0x55c10988d840 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.270+0000 7f29301c4740 10 client.0 _ll_register_callbacks cb 0x55c1097e7ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109897810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109897810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40444/0 (socket says 172.21.15.16:40444) 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 -- 172.21.15.16:0/2304068871 learned_addr learned my addr 172.21.15.16:0/2304068871 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f2926ffd640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c10988ce50 0x55c1098972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f29277fe640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1098929b0 0x55c10989e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 -- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c10988ce50 msgr2=0x55c1098972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c10988ce50 0x55c1098972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 -- 172.21.15.16:0/2304068871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1098929b0 msgr2=0x55c10989e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1098929b0 0x55c10989e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f291ffff640 1 -- 172.21.15.16:0/2304068871 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1095d1620 con 0x55c10988d840 2024-04-25T01:30:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.271+0000 7f29277fe640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1098929b0 0x55c10989e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f291ffff640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109897810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2910006a50 tx=0x7f2910014d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e9f34c77222b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f290e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:30:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f290e7fc640 1 -- 172.21.15.16:0/2304068871 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910029060 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f290e7fc640 1 -- 172.21.15.16:0/2304068871 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2910004450 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2910013920 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f290e7fc640 1 -- 172.21.15.16:0/2304068871 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2910017050 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f29301c4740 10 client.25129 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c1098a1bc0 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.272+0000 7f290e7fc640 1 -- 172.21.15.16:0/2304068871 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2910013920 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f290e7fc640 1 -- 172.21.15.16:0/2304068871 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f291001c610 con 0x55c10988d840 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f29301c4740 10 client.25129 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f29301c4740 10 client.25129 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f29301c4740 20 client.25129 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f29301c4740 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1098a2fd0 0x55c1098a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f29301c4740 4 client.25129 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1097fe1f0 con 0x55c1098a2fd0 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.273+0000 7f2926ffd640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1098a2fd0 0x55c1098a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.274+0000 7f2926ffd640 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1098a2fd0 0x55c1098a53f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c27e2cd8cb9736 server_cookie=b7ca98dacccb0a05 in_seq=0 out_seq=0 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.274+0000 7f290e7fc640 10 client.25129 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.274+0000 7f290e7fc640 1 -- 172.21.15.16:0/2304068871 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1097fe1f0 con 0x55c1098a2fd0 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f290e7fc640 10 client.25129 handle_command_reply: tid=0 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 client.25129 shutdown 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 msgr2=0x55c109897810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109897810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2910006a50 tx=0x7f2910014d10 comp rx=0 tx=0).stop 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 shutdown_connections 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1098929b0 0x55c10989e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c109897810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1098a2fd0 0x55c1098a53f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 --2- 172.21.15.16:0/2304068871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c10988ce50 0x55c1098972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 >> 172.21.15.16:0/2304068871 conn(0x55c1098884c0 msgr2=0x55c1098875c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:00.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f290e7fc640 0 client.25129 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:00.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 shutdown_connections 2024-04-25T01:30:00.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 1 -- 172.21.15.16:0/2304068871 wait complete. 2024-04-25T01:30:00.258 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.275+0000 7f29301c4740 20 client.25129 trim_cache size 0 max 16384 2024-04-25T01:30:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.303+0000 7f29301c4740 1 Processor -- start 2024-04-25T01:30:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.303+0000 7f29301c4740 1 -- start start 2024-04-25T01:30:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29301c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988ce50 0x55c109896cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29301c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 0x55c10989e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29301c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109886d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29301c4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2908010dc0 con 0x55c10988d840 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29301c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f290800e690 con 0x55c1098929b0 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29301c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f290800c4e0 con 0x55c10988ce50 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f291ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 0x55c10989e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f291ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 0x55c10989e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38780/0 (socket says 172.21.15.16:38780) 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f291ffff640 1 -- 172.21.15.16:0/3190455564 learned_addr learned my addr 172.21.15.16:0/3190455564 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988ce50 0x55c109896cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f29277fe640 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109886d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f2926ffd640 1 -- 172.21.15.16:0/3190455564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 msgr2=0x55c109886d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109886d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f2926ffd640 1 -- 172.21.15.16:0/3190455564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 msgr2=0x55c10989e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 0x55c10989e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f2926ffd640 1 -- 172.21.15.16:0/3190455564 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10988e0d0 con 0x55c10988ce50 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.304+0000 7f291ffff640 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 0x55c10989e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.305+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988ce50 0x55c109896cb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55c10988e2b0 tx=0x7f29180dc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2b161a8ac42dcd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.305+0000 7f291cff9640 1 -- 172.21.15.16:0/3190455564 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918049040 con 0x55c10988ce50 2024-04-25T01:30:00.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.305+0000 7f291cff9640 1 -- 172.21.15.16:0/3190455564 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29180bc7e0 con 0x55c10988ce50 2024-04-25T01:30:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.305+0000 7f291cff9640 1 -- 172.21.15.16:0/3190455564 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180ba3e0 con 0x55c10988ce50 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 -- 172.21.15.16:0/3190455564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988ce50 msgr2=0x55c109896cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988ce50 0x55c109896cb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55c10988e2b0 tx=0x7f29180dc040 comp rx=0 tx=0).stop 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 -- 172.21.15.16:0/3190455564 shutdown_connections 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c109886d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988d840 0x55c10989e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 --2- 172.21.15.16:0/3190455564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988ce50 0x55c109896cb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 -- 172.21.15.16:0/3190455564 >> 172.21.15.16:0/3190455564 conn(0x55c1098884c0 msgr2=0x55c1098965a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:00.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 -- 172.21.15.16:0/3190455564 shutdown_connections 2024-04-25T01:30:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.307+0000 7f29301c4740 1 -- 172.21.15.16:0/3190455564 wait complete. 2024-04-25T01:30:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.308+0000 7f29301c4740 1 -- start start 2024-04-25T01:30:00.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.308+0000 7f29301c4740 1 Processor -- start 2024-04-25T01:30:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.308+0000 7f29301c4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10989bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.308+0000 7f29301c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c10989c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10989bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10989bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38782/0 (socket says 172.21.15.16:38782) 2024-04-25T01:30:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f291ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c10989c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f291ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c10989c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40460/0 (socket says 172.21.15.16:40460) 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 -- 172.21.15.16:0/3979351153 learned_addr learned my addr 172.21.15.16:0/3979351153 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c10989d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f290800cb50 con 0x55c10988ce50 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f290800dfc0 con 0x55c1098929b0 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f290800d220 con 0x55c10988d840 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f29277fe640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c10989d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f29301c4740 10 client.0 _ll_register_callbacks cb 0x7f29200076d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 -- 172.21.15.16:0/3979351153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 msgr2=0x55c10989c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c10989c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 -- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 msgr2=0x55c10989d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c10989d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f2926ffd640 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10989dac0 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.309+0000 7f291ffff640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c10989c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f2926ffd640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10989bf80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2918000c00 tx=0x7f2918009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f524597276aaa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f290e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f290e7fc640 1 -- 172.21.15.16:0/3979351153 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180d4070 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f290e7fc640 1 -- 172.21.15.16:0/3979351153 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f29180dc0b0 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29180bc7e0 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f290e7fc640 1 -- 172.21.15.16:0/3979351153 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2918049040 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f29301c4740 10 client.15492 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.310+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c109909530 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f290e7fc640 1 -- 172.21.15.16:0/3979351153 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29180bc0e0 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f290e7fc640 1 -- 172.21.15.16:0/3979351153 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f29180c48c0 con 0x55c10988ce50 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f29301c4740 10 client.15492 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f29301c4740 10 client.15492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f29301c4740 20 client.15492 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c109909eb0 0x55c10990c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f29301c4740 4 client.15492 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:30:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f29180bc7e0 con 0x55c109909eb0 2024-04-25T01:30:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.311+0000 7f291ffff640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c109909eb0 0x55c10990c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.312+0000 7f290e7fc640 10 client.15492 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.312+0000 7f291ffff640 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c109909eb0 0x55c10990c2d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26cbe4490cc20332 server_cookie=48c30e7fd8cec096 in_seq=0 out_seq=0 2024-04-25T01:30:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f290e7fc640 1 -- 172.21.15.16:0/3979351153 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f29180bc7e0 con 0x55c109909eb0 2024-04-25T01:30:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f290e7fc640 10 client.15492 handle_command_reply: tid=0 2024-04-25T01:30:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 client.15492 shutdown 2024-04-25T01:30:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 msgr2=0x55c10989bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10989bf80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2918000c00 tx=0x7f2918009680 comp rx=0 tx=0).stop 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 shutdown_connections 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c109909eb0 0x55c10990c2d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1098929b0 0x55c10989d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c10988d840 0x55c10989c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f290e7fc640 0 client.15492 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 --2- 172.21.15.16:0/3979351153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c10988ce50 0x55c10989bf80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 >> 172.21.15.16:0/3979351153 conn(0x55c1098884c0 msgr2=0x55c109886d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 shutdown_connections 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.317+0000 7f29301c4740 1 -- 172.21.15.16:0/3979351153 wait complete. 2024-04-25T01:30:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f29301c4740 20 client.15492 trim_cache size 0 max 16384 2024-04-25T01:30:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:30:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:30:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:30:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:30:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29140a2d30 msgr2=0x7f29140a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29140a2d30 0x7f29140a51f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55c10988e0a0 tx=0x7f2920004ab0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 msgr2=0x7f292819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 0x7f292819dff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f29180314e0 tx=0x7f2918003d50 comp rx=0 tx=0).stop 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 shutdown_connections 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f29140a2d30 0x7f29140a51f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2928103380 0x7f29281a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29281029c0 0x7f292819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 --2- 172.21.15.16:0/2030792818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29281017d0 0x7f292819dff0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.318+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 >> 172.21.15.16:0/2030792818 conn(0x7f29280fcd60 msgr2=0x7f292810c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.319+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 shutdown_connections 2024-04-25T01:30:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:00.319+0000 7f292d3d5640 1 -- 172.21.15.16:0/2030792818 wait complete. 2024-04-25T01:30:00.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:00 smithi016 ceph-mon[29153]: overall HEALTH_OK 2024-04-25T01:30:00.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:00 smithi038 ceph-mon[32841]: overall HEALTH_OK 2024-04-25T01:30:00.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:00 smithi156 ceph-mon[32631]: overall HEALTH_OK 2024-04-25T01:30:01.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:01 smithi016 ceph-mon[29153]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T01:30:01.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:01 smithi038 ceph-mon[32841]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T01:30:01.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:01 smithi156 ceph-mon[32631]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T01:30:03.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:03 smithi016 ceph-mon[29153]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:30:03.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:03 smithi038 ceph-mon[32841]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:30:03.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:03 smithi156 ceph-mon[32631]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:30:05.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:05 smithi016 ceph-mon[29153]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-25T01:30:05.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:05 smithi038 ceph-mon[32841]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-25T01:30:05.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:05 smithi156 ceph-mon[32631]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-25T01:30:07.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:07 smithi016 ceph-mon[29153]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-04-25T01:30:07.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:07 smithi038 ceph-mon[32841]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-04-25T01:30:07.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:07 smithi156 ceph-mon[32631]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-04-25T01:30:09.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:09 smithi016 ceph-mon[29153]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T01:30:09.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:09 smithi038 ceph-mon[32841]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T01:30:09.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:09 smithi156 ceph-mon[32631]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T01:30:11.800 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:11 smithi156 ceph-mon[32631]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-25T01:30:11.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:11 smithi016 ceph-mon[29153]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-25T01:30:11.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:11 smithi038 ceph-mon[32841]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 -- 172.21.15.16:0/2117691446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc1006e0 msgr2=0x7fcfbc100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 --2- 172.21.15.16:0/2117691446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc100b00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009900 tx=0x7fcfb802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 -- 172.21.15.16:0/2117691446 shutdown_connections 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 --2- 172.21.15.16:0/2117691446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc102290 0x7fcfbc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 --2- 172.21.15.16:0/2117691446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1018d0 0x7fcfbc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 --2- 172.21.15.16:0/2117691446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc100b00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 -- 172.21.15.16:0/2117691446 >> 172.21.15.16:0/2117691446 conn(0x7fcfbc0fbc50 msgr2=0x7fcfbc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.465+0000 7fcfc3763640 1 -- 172.21.15.16:0/2117691446 shutdown_connections 2024-04-25T01:30:13.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.466+0000 7fcfc3763640 1 -- 172.21.15.16:0/2117691446 wait complete. 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.466+0000 7fcfc3763640 1 Processor -- start 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.466+0000 7fcfc3763640 1 -- start start 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.466+0000 7fcfc3763640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc3763640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc1018d0 0x7fcfbc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc3763640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc102290 0x7fcfbc19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc3763640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc10c1e0 con 0x7fcfbc1018d0 2024-04-25T01:30:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc3763640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc10c060 con 0x7fcfbc102290 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc3763640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc10c360 con 0x7fcfbc1006e0 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc0cfe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc1018d0 0x7fcfbc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:53484/0 (socket says 172.21.15.16:53484) 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1784266595 learned_addr learned my addr 172.21.15.16:0/1784266595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc102290 0x7fcfbc19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 -- 172.21.15.16:0/1784266595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 msgr2=0x7fcfbc197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 -- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc1018d0 msgr2=0x7fcfbc197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc1018d0 0x7fcfbc197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 -- 172.21.15.16:0/1784266595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc19f590 con 0x7fcfbc102290 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc1018d0 0x7fcfbc197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:13.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.467+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc102290 0x7fcfbc19ef20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009fd0 tx=0x7fcfb400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.468+0000 7fcfae7fc640 1 -- 172.21.15.16:0/1784266595 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400ce20 con 0x7fcfbc102290 2024-04-25T01:30:13.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.468+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc19f880 con 0x7fcfbc102290 2024-04-25T01:30:13.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.468+0000 7fcfae7fc640 1 -- 172.21.15.16:0/1784266595 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb4005020 con 0x7fcfbc102290 2024-04-25T01:30:13.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.468+0000 7fcfae7fc640 1 -- 172.21.15.16:0/1784266595 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4010690 con 0x7fcfbc102290 2024-04-25T01:30:13.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.468+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc19fdd0 con 0x7fcfbc102290 2024-04-25T01:30:13.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.470+0000 7fcfae7fc640 1 -- 172.21.15.16:0/1784266595 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fcfb40051d0 con 0x7fcfbc102290 2024-04-25T01:30:13.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.471+0000 7fcfc647b740 1 Processor -- start 2024-04-25T01:30:13.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.471+0000 7fcfc647b740 1 -- start start 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfae7fc640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcfa80a2d30 0x7fcfa80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 0x55c1d5e1e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e23750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e26250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc647b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1d5d7dcb0 con 0x55c1d5e1e260 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc647b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1d5d265b0 con 0x55c1d5e1ec50 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc647b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c1d5d79c70 con 0x55c1d5e23dc0 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcfa80a2d30 0x7fcfa80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 0x55c1d5e1e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e26250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 0x55c1d5e1e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37850/0 (socket says 172.21.15.16:37850) 2024-04-25T01:30:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc14ff640 1 -- 172.21.15.16:0/521983416 learned_addr learned my addr 172.21.15.16:0/521983416 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e23750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfae7fc640 1 -- 172.21.15.16:0/1784266595 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcfb40cf700 con 0x7fcfbc102290 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 -- 172.21.15.16:0/521983416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 msgr2=0x55c1d5e23750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e23750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 -- 172.21.15.16:0/521983416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 msgr2=0x55c1d5e1e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 0x55c1d5e1e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 -- 172.21.15.16:0/521983416 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d5a92210 con 0x55c1d5e23dc0 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 0x55c1d5e1e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e26250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c760 tx=0x7fcfb407fcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22463499d5366753 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.472+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcfa80a2d30 0x7fcfa80a51f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55c1d5e1f050 tx=0x7fcfb000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfa77fe640 1 -- 172.21.15.16:0/521983416 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4081040 con 0x55c1d5e23dc0 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfa77fe640 1 -- 172.21.15.16:0/521983416 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb40a0290 con 0x55c1d5e23dc0 2024-04-25T01:30:13.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfa77fe640 1 -- 172.21.15.16:0/521983416 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb409a270 con 0x55c1d5e23dc0 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 -- 172.21.15.16:0/521983416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 msgr2=0x55c1d5e26250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e26250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c760 tx=0x7fcfb407fcd0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 -- 172.21.15.16:0/521983416 shutdown_connections 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e26250 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e23750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 --2- 172.21.15.16:0/521983416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1e260 0x55c1d5e1e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.473+0000 7fcfc647b740 1 -- 172.21.15.16:0/521983416 >> 172.21.15.16:0/521983416 conn(0x55c1d5e154c0 msgr2=0x55c1d5e158c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.474+0000 7fcfc647b740 1 -- 172.21.15.16:0/521983416 shutdown_connections 2024-04-25T01:30:13.454 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.474+0000 7fcfc647b740 1 -- 172.21.15.16:0/521983416 wait complete. 2024-04-25T01:30:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.474+0000 7fcfc647b740 1 -- start start 2024-04-25T01:30:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.474+0000 7fcfc647b740 1 Processor -- start 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.474+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e286f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e28c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc647b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c1d5e26d70 con 0x55c1d5e1ec50 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc647b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1d5d39b50 con 0x55c1d5e23dc0 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc647b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c1d5e26ef0 con 0x55c1d5e1e260 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e286f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:53514/0 (socket says 172.21.15.16:53514) 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1269835019 learned_addr learned my addr 172.21.15.16:0/1269835019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:13.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc647b740 10 client.0 _ll_register_callbacks cb 0x55c1d5d74ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e28c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e23dc0 msgr2=0x55c1d5e2fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1269835019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 msgr2=0x55c1d5e28c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e28c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1269835019 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d5b5e620 con 0x55c1d5e1e260 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e28c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.475+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e286f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009230 tx=0x7fcfb8005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc3fe7c69e16a2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1269835019 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80055a0 con 0x55c1d5e1e260 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1269835019 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcfb80046c0 con 0x55c1d5e1e260 2024-04-25T01:30:13.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb40a0290 con 0x55c1d5e1e260 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1269835019 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcfb80048c0 con 0x55c1d5e1e260 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfc647b740 10 client.25147 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c1d5e33030 con 0x55c1d5e1e260 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.476+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1269835019 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb40a0290 con 0x55c1d5e1e260 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1269835019 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcfb803baa0 con 0x55c1d5e1e260 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc647b740 10 client.25147 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc647b740 10 client.25147 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc647b740 20 client.25147 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e34760 0x55c1d5e36bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc647b740 4 client.25147 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1d5d8b1f0 con 0x55c1d5e34760 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.477+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e34760 0x55c1d5e36bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.478+0000 7fcfa4ff9640 10 client.25147 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.478+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e34760 0x55c1d5e36bf0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b97d018696a364 server_cookie=e0a381c01aac7a0f in_seq=0 out_seq=0 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.478+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1269835019 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1d5d8b1f0 con 0x55c1d5e34760 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.478+0000 7fcfa4ff9640 10 client.25147 handle_command_reply: tid=0 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.478+0000 7fcfc647b740 1 client.25147 shutdown 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 msgr2=0x55c1d5e286f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e286f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009230 tx=0x7fcfb8005270 comp rx=0 tx=0).stop 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 shutdown_connections 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e34760 0x55c1d5e36bf0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e28c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1269835019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e286f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 >> 172.21.15.16:0/1269835019 conn(0x55c1d5e154c0 msgr2=0x55c1d5e145c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfa4ff9640 0 client.25147 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.479+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 shutdown_connections 2024-04-25T01:30:13.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.480+0000 7fcfc647b740 1 -- 172.21.15.16:0/1269835019 wait complete. 2024-04-25T01:30:13.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.480+0000 7fcfc647b740 20 client.25147 trim_cache size 0 max 16384 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 Processor -- start 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 -- start start 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e27d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e28250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a9220 con 0x55c1d5e23dc0 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a6af0 con 0x55c1d5e1ec50 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.502+0000 7fcfc647b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a4940 con 0x55c1d5e1e260 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e27d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc1d00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e28250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e27d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:53528/0 (socket says 172.21.15.16:53528) 2024-04-25T01:30:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc1d00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e28250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37876/0 (socket says 172.21.15.16:37876) 2024-04-25T01:30:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc14ff640 1 -- 172.21.15.16:0/2335510083 learned_addr learned my addr 172.21.15.16:0/2335510083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 -- 172.21.15.16:0/2335510083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 msgr2=0x55c1d5e27d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e27d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 -- 172.21.15.16:0/2335510083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e23dc0 msgr2=0x55c1d5e28250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e28250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 -- 172.21.15.16:0/2335510083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d5e16230 con 0x55c1d5e1ec50 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.503+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2fb90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55c1d5e37470 tx=0x7fcfb0014070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bda05a37de878be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.504+0000 7fcfa6ffd640 1 -- 172.21.15.16:0/2335510083 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0012650 con 0x55c1d5e1ec50 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.504+0000 7fcfa6ffd640 1 -- 172.21.15.16:0/2335510083 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb0012110 con 0x55c1d5e1ec50 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.504+0000 7fcfa6ffd640 1 -- 172.21.15.16:0/2335510083 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0029810 con 0x55c1d5e1ec50 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 -- 172.21.15.16:0/2335510083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 msgr2=0x55c1d5e2fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2fb90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55c1d5e37470 tx=0x7fcfb0014070 comp rx=0 tx=0).stop 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 -- 172.21.15.16:0/2335510083 shutdown_connections 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e28250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2fb90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 --2- 172.21.15.16:0/2335510083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e1e260 0x55c1d5e27d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 -- 172.21.15.16:0/2335510083 >> 172.21.15.16:0/2335510083 conn(0x55c1d5e154c0 msgr2=0x55c1d5e14440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 -- 172.21.15.16:0/2335510083 shutdown_connections 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 -- 172.21.15.16:0/2335510083 wait complete. 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.505+0000 7fcfc647b740 1 -- start start 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 Processor -- start 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 0x55c1d5e2d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a4fb0 con 0x55c1d5e1ec50 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a6420 con 0x55c1d5e1e260 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a5680 con 0x55c1d5e23dc0 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc647b740 10 client.0 _ll_register_callbacks cb 0x7fcfb8030ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 0x55c1d5e2d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc14ff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 0x55c1d5e2d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:48212/0 (socket says 172.21.15.16:48212) 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1000140140 learned_addr learned my addr 172.21.15.16:0/1000140140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.506+0000 7fcfc1d00640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1000140140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 msgr2=0x55c1d5e2ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1000140140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 msgr2=0x55c1d5e2dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc14ff640 1 -- 172.21.15.16:0/1000140140 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d5e1dc00 con 0x55c1d5e1e260 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfc14ff640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 0x55c1d5e2d710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80444a0 tx=0x7fcfb8005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c635daa55a76fe7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.507+0000 7fcfa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1000140140 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8053070 con 0x55c1d5e1e260 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1000140140 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcfb8044bd0 con 0x55c1d5e1e260 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb0012110 con 0x55c1d5e1e260 2024-04-25T01:30:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1000140140 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcfb8044dd0 con 0x55c1d5e1e260 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 10 client.25178 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c1d5e9f560 con 0x55c1d5e1e260 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1000140140 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb0012110 con 0x55c1d5e1e260 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1000140140 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcfb803b410 con 0x55c1d5e1e260 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 10 client.25178 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 10 client.25178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 20 client.25178 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.508+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e9fe20 0x55c1d5ea2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.509+0000 7fcfc647b740 4 client.25178 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.509+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1d5d85540 con 0x55c1d5e9fe20 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.509+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e9fe20 0x55c1d5ea2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.509+0000 7fcfc0cfe640 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e9fe20 0x55c1d5ea2240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89b92bd790f84b9b server_cookie=f947b25abf01f124 in_seq=0 out_seq=0 2024-04-25T01:30:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.509+0000 7fcfa4ff9640 10 client.25178 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfa4ff9640 1 -- 172.21.15.16:0/1000140140 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1d5d85540 con 0x55c1d5e9fe20 2024-04-25T01:30:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfa4ff9640 10 client.25178 handle_command_reply: tid=0 2024-04-25T01:30:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 client.25178 shutdown 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 msgr2=0x55c1d5e2d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 0x55c1d5e2d710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80444a0 tx=0x7fcfb8005d10 comp rx=0 tx=0).stop 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 shutdown_connections 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1d5e23dc0 0x55c1d5e2ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c1d5e1ec50 0x55c1d5e2dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c1d5e9fe20 0x55c1d5ea2240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 --2- 172.21.15.16:0/1000140140 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1d5e1e260 0x55c1d5e2d710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfa4ff9640 0 client.25178 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 >> 172.21.15.16:0/1000140140 conn(0x55c1d5e154c0 msgr2=0x55c1d5e18440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 shutdown_connections 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.513+0000 7fcfc647b740 1 -- 172.21.15.16:0/1000140140 wait complete. 2024-04-25T01:30:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc647b740 20 client.25178 trim_cache size 0 max 16384 2024-04-25T01:30:13.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:30:13.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:30:13.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:30:13.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcfa80a2d30 msgr2=0x7fcfa80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcfa80a2d30 0x7fcfa80a51f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55c1d5e1f050 tx=0x7fcfb000b390 comp rx=0 tx=0).stop 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc102290 msgr2=0x7fcfbc19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc102290 0x7fcfbc19ef20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009fd0 tx=0x7fcfb400ef10 comp rx=0 tx=0).stop 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 shutdown_connections 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fcfa80a2d30 0x7fcfa80a51f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfbc102290 0x7fcfbc19ef20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfbc1018d0 0x7fcfbc197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 --2- 172.21.15.16:0/1784266595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfbc1006e0 0x7fcfbc197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.514+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 >> 172.21.15.16:0/1784266595 conn(0x7fcfbc0fbc50 msgr2=0x7fcfbc0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.515+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 shutdown_connections 2024-04-25T01:30:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:13.515+0000 7fcfc3763640 1 -- 172.21.15.16:0/1784266595 wait complete. 2024-04-25T01:30:13.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:13 smithi016 ceph-mon[29153]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:30:13.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:13 smithi038 ceph-mon[32841]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:30:14.000 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:13 smithi156 ceph-mon[32631]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:30:15.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:15 smithi016 ceph-mon[29153]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-04-25T01:30:15.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:15 smithi038 ceph-mon[32841]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-04-25T01:30:15.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:15 smithi156 ceph-mon[32631]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-04-25T01:30:17.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:17 smithi016 ceph-mon[29153]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:30:17.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:17 smithi038 ceph-mon[32841]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:30:17.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:17 smithi156 ceph-mon[32631]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:30:19.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:19 smithi016 ceph-mon[29153]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-25T01:30:19.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:19 smithi038 ceph-mon[32841]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-25T01:30:19.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:19 smithi156 ceph-mon[32631]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-25T01:30:21.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:21 smithi016 ceph-mon[29153]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:30:21.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:21 smithi038 ceph-mon[32841]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:30:21.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:21 smithi156 ceph-mon[32631]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:30:23.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:23 smithi016 ceph-mon[29153]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:30:23.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:23 smithi038 ceph-mon[32841]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:30:23.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:23 smithi156 ceph-mon[32631]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:30:25.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:25 smithi016 ceph-mon[29153]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T01:30:25.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:25 smithi038 ceph-mon[32841]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T01:30:25.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:25 smithi156 ceph-mon[32631]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T01:30:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.662+0000 7f77d1bc8640 1 -- 172.21.15.16:0/782435534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1018b0 msgr2=0x7f77cc101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.662+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/782435534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1018b0 0x7f77cc101d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009860 tx=0x7f77c002a340 comp rx=0 tx=0).stop 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 -- 172.21.15.16:0/782435534 shutdown_connections 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/782435534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 0x7f77cc108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/782435534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1018b0 0x7f77cc101d30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/782435534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1006c0 0x7f77cc100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 -- 172.21.15.16:0/782435534 >> 172.21.15.16:0/782435534 conn(0x7f77cc0fbc30 msgr2=0x7f77cc0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 -- 172.21.15.16:0/782435534 shutdown_connections 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.663+0000 7f77d1bc8640 1 -- 172.21.15.16:0/782435534 wait complete. 2024-04-25T01:30:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 Processor -- start 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 -- start start 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 0x7f77cc199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1018b0 0x7f77cc19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 0x7f77cc1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10d2d0 con 0x7f77cc102270 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10d150 con 0x7f77cc1006c0 2024-04-25T01:30:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.664+0000 7f77d1bc8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10d450 con 0x7f77cc1018b0 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77cb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 0x7f77cc199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77cb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 0x7f77cc199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46250/0 (socket says 172.21.15.16:46250) 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77cb7fe640 1 -- 172.21.15.16:0/2644155702 learned_addr learned my addr 172.21.15.16:0/2644155702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1018b0 0x7f77cc19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77cbfff640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 0x7f77cc1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 -- 172.21.15.16:0/2644155702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 msgr2=0x7f77cc199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 0x7f77cc199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 -- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 msgr2=0x7f77cc1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 0x7f77cc1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 -- 172.21.15.16:0/2644155702 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc1a1750 con 0x7f77cc1018b0 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77cbfff640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 0x7f77cc1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 0x7f77cc199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.665+0000 7f77caffd640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1018b0 0x7f77cc19a070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f77c002a860 tx=0x7f77c0030e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.666+0000 7f77c8ff9640 1 -- 172.21.15.16:0/2644155702 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0040020 con 0x7f77cc1018b0 2024-04-25T01:30:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.666+0000 7f77c8ff9640 1 -- 172.21.15.16:0/2644155702 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77c0038040 con 0x7f77cc1018b0 2024-04-25T01:30:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.666+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77cc1a19e0 con 0x7f77cc1018b0 2024-04-25T01:30:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.666+0000 7f77c8ff9640 1 -- 172.21.15.16:0/2644155702 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0037070 con 0x7f77cc1018b0 2024-04-25T01:30:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.666+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77cc1a1ed0 con 0x7f77cc1018b0 2024-04-25T01:30:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77c8ff9640 1 -- 172.21.15.16:0/2644155702 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f77c0043020 con 0x7f77cc1018b0 2024-04-25T01:30:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 Processor -- start 2024-04-25T01:30:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 -- start start 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x558874065270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874065840 0x55887406a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x558874065270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x558874065270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39156/0 (socket says 172.21.15.16:39156) 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x55887406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77cbfff640 1 -- 172.21.15.16:0/443139230 learned_addr learned my addr 172.21.15.16:0/443139230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558873fc8cb0 con 0x558874064e50 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558873f715b0 con 0x55887406a9b0 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.670+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558873fc4c70 con 0x558874065840 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77caffd640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x55887406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874065840 0x55887406a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77caffd640 1 -- 172.21.15.16:0/443139230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874065840 msgr2=0x55887406a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77caffd640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874065840 0x55887406a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77caffd640 1 -- 172.21.15.16:0/443139230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 msgr2=0x558874065270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77caffd640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x558874065270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77caffd640 1 -- 172.21.15.16:0/443139230 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558873cdd210 con 0x55887406a9b0 2024-04-25T01:30:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77cbfff640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x558874065270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874065840 0x55887406a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77c8ff9640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f77b40a2da0 0x7f77b40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.671+0000 7f77cbfff640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f77b40a2da0 0x7f77b40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.672+0000 7f77caffd640 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x55887406ce40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f77c00ec500 tx=0x7f77c00f2ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf1f0320bd95362 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.672+0000 7f77b1ffb640 1 -- 172.21.15.16:0/443139230 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c00b75b0 con 0x55887406a9b0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.672+0000 7f77b1ffb640 1 -- 172.21.15.16:0/443139230 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77c00c5a10 con 0x55887406a9b0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.672+0000 7f77b1ffb640 1 -- 172.21.15.16:0/443139230 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c00c8770 con 0x55887406a9b0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.672+0000 7f77c8ff9640 1 -- 172.21.15.16:0/2644155702 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f77c00eea50 con 0x7f77cc1018b0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 msgr2=0x55887406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77cbfff640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f77b40a2da0 0x7f77b40a5260 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x5588740660a0 tx=0x7f77c4010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x55887406ce40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f77c00ec500 tx=0x7f77c00f2ab0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 shutdown_connections 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x55887406ce40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874065840 0x55887406a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 --2- 172.21.15.16:0/443139230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x558874065270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 >> 172.21.15.16:0/443139230 conn(0x5588740604c0 msgr2=0x5588740608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:26.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 shutdown_connections 2024-04-25T01:30:26.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 -- 172.21.15.16:0/443139230 wait complete. 2024-04-25T01:30:26.653 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.673+0000 7f77d49b2740 1 -- start start 2024-04-25T01:30:26.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 Processor -- start 2024-04-25T01:30:26.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x55887406edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x55887406f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 0x558874076340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558873fc8cb0 con 0x55887406a9b0 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558873f84b50 con 0x558874065840 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.674+0000 7f77d49b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558873f715b0 con 0x558874064e50 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77d49b2740 10 client.0 _ll_register_callbacks cb 0x558873fbfca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77caffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x55887406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 0x558874076340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 0x558874076340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39162/0 (socket says 172.21.15.16:39162) 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 -- 172.21.15.16:0/1800750397 learned_addr learned my addr 172.21.15.16:0/1800750397 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x55887406edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 -- 172.21.15.16:0/1800750397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 msgr2=0x55887406edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x55887406edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 -- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 msgr2=0x55887406f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x55887406f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cbfff640 1 -- 172.21.15.16:0/1800750397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558873da9620 con 0x55887406a9b0 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.675+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x55887406edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77cbfff640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 0x558874076340 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f77c4003ef0 tx=0x7f77c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3975543da606d229 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77aaffd640 1 -- 172.21.15.16:0/1800750397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4004d20 con 0x55887406a9b0 2024-04-25T01:30:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77aaffd640 1 -- 172.21.15.16:0/1800750397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f77c4018670 con 0x55887406a9b0 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77c00c5a10 con 0x55887406a9b0 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77aaffd640 1 -- 172.21.15.16:0/1800750397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f77c4018870 con 0x55887406a9b0 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77d49b2740 10 client.15516 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.676+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558874079720 con 0x55887406a9b0 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.677+0000 7f77aaffd640 1 -- 172.21.15.16:0/1800750397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77c4005020 con 0x55887406a9b0 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.677+0000 7f77aaffd640 1 -- 172.21.15.16:0/1800750397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f77c400b040 con 0x55887406a9b0 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.677+0000 7f77d49b2740 10 client.15516 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.677+0000 7f77d49b2740 10 client.15516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.678+0000 7f77d49b2740 20 client.15516 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.678+0000 7f77d49b2740 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55887407ae80 0x55887407d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.678+0000 7f77d49b2740 4 client.15516 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.678+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55887407ae80 0x55887407d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.679+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f77c00c5a10 con 0x55887407ae80 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.679+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55887407ae80 0x55887407d270 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=186b32fa6a13469c server_cookie=e8f604af3b86a8ae in_seq=0 out_seq=0 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.679+0000 7f77aaffd640 10 client.15516 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.680+0000 7f77aaffd640 1 -- 172.21.15.16:0/1800750397 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f77c00c5a10 con 0x55887407ae80 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.680+0000 7f77aaffd640 10 client.15516 handle_command_reply: tid=0 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.680+0000 7f77d49b2740 1 client.15516 shutdown 2024-04-25T01:30:26.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.680+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 msgr2=0x558874076340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.680+0000 7f77d49b2740 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 0x558874076340 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f77c4003ef0 tx=0x7f77c4004ab0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.680+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 shutdown_connections 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55887406a9b0 0x558874076340 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55887407ae80 0x55887407d270 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x55887406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77aaffd640 0 client.15516 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 --2- 172.21.15.16:0/1800750397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x55887406edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 >> 172.21.15.16:0/1800750397 conn(0x5588740604c0 msgr2=0x55887405f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 shutdown_connections 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 1 -- 172.21.15.16:0/1800750397 wait complete. 2024-04-25T01:30:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.681+0000 7f77d49b2740 20 client.15516 trim_cache size 0 max 16384 2024-04-25T01:30:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.707+0000 7f77d49b2740 1 Processor -- start 2024-04-25T01:30:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.707+0000 7f77d49b2740 1 -- start start 2024-04-25T01:30:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.707+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x55887405ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.707+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x558874076300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.707+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55887406a9b0 0x55887405f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77d49b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc1ad7a0 con 0x558874064e50 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77d49b2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc1ab070 con 0x558874065840 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77d49b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc1a8ec0 con 0x55887406a9b0 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55887406a9b0 0x55887405f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55887406a9b0 0x55887405f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:33846/0 (socket says 172.21.15.16:33846) 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77cbfff640 1 -- 172.21.15.16:0/2972948591 learned_addr learned my addr 172.21.15.16:0/2972948591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x55887405ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77caffd640 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x558874076300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77caffd640 1 -- 172.21.15.16:0/2972948591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55887406a9b0 msgr2=0x55887405f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77caffd640 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55887406a9b0 0x55887405f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77caffd640 1 -- 172.21.15.16:0/2972948591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 msgr2=0x55887405ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77caffd640 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x55887405ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.708+0000 7f77caffd640 1 -- 172.21.15.16:0/2972948591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55887407d8f0 con 0x558874065840 2024-04-25T01:30:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.709+0000 7f77caffd640 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x558874076300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77c00cd240 tx=0x7f77c0032dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d4d2d2bd66eef34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.709+0000 7f77b17fa640 1 -- 172.21.15.16:0/2972948591 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c00c8990 con 0x558874065840 2024-04-25T01:30:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.709+0000 7f77b17fa640 1 -- 172.21.15.16:0/2972948591 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77c00bba10 con 0x558874065840 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.709+0000 7f77b17fa640 1 -- 172.21.15.16:0/2972948591 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c00c8bb0 con 0x558874065840 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 -- 172.21.15.16:0/2972948591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 msgr2=0x558874076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x558874076300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77c00cd240 tx=0x7f77c0032dd0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 -- 172.21.15.16:0/2972948591 shutdown_connections 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55887406a9b0 0x55887405f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558874065840 0x558874076300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2972948591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874064e50 0x55887405ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 -- 172.21.15.16:0/2972948591 >> 172.21.15.16:0/2972948591 conn(0x5588740604c0 msgr2=0x5588740608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 -- 172.21.15.16:0/2972948591 shutdown_connections 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.710+0000 7f77d49b2740 1 -- 172.21.15.16:0/2972948591 wait complete. 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.711+0000 7f77d49b2740 1 -- start start 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.711+0000 7f77d49b2740 1 Processor -- start 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.711+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x558874074050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.711+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 0x558874074590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77d49b2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x558874075530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77d49b2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc1a9530 con 0x558874065840 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77d49b2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc1aa9a0 con 0x55887406a9b0 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77d49b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc1a9c00 con 0x558874064e50 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 0x558874074590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77caffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 0x558874074590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39176/0 (socket says 172.21.15.16:39176) 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77caffd640 1 -- 172.21.15.16:0/2992197526 learned_addr learned my addr 172.21.15.16:0/2992197526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x558874074050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77d49b2740 10 client.0 _ll_register_callbacks cb 0x7f77c400e990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cbfff640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x558874075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 -- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 msgr2=0x558874075530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x558874075530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 -- 172.21.15.16:0/2992197526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 msgr2=0x558874074590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 0x558874074590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 -- 172.21.15.16:0/2992197526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558874075b20 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77caffd640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 0x558874074590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77cb7fe640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x558874074050 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55887407db40 tx=0x7f77bc010e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbdaa60e7030ddb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.712+0000 7f77aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77aaffd640 1 -- 172.21.15.16:0/2992197526 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00fd80 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77aaffd640 1 -- 172.21.15.16:0/2992197526 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f77bc005020 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77c00bba10 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77aaffd640 1 -- 172.21.15.16:0/2992197526 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f77bc030050 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77d49b2740 10 client.25171 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5588740ea2f0 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.713+0000 7f77aaffd640 1 -- 172.21.15.16:0/2992197526 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77c00bba10 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77aaffd640 1 -- 172.21.15.16:0/2992197526 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f77bc020070 con 0x558874064e50 2024-04-25T01:30:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77d49b2740 10 client.25171 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77d49b2740 10 client.25171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77d49b2740 20 client.25171 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5588740eaca0 0x5588740ed0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77d49b2740 4 client.25171 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558873fd61f0 con 0x5588740eaca0 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77caffd640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5588740eaca0 0x5588740ed0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77caffd640 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5588740eaca0 0x5588740ed0c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6993b21518d3ed9c server_cookie=80c190d545132c2f in_seq=0 out_seq=0 2024-04-25T01:30:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.714+0000 7f77aaffd640 10 client.25171 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.718+0000 7f77aaffd640 1 -- 172.21.15.16:0/2992197526 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558873fd61f0 con 0x5588740eaca0 2024-04-25T01:30:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.718+0000 7f77aaffd640 10 client.25171 handle_command_reply: tid=0 2024-04-25T01:30:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.718+0000 7f77d49b2740 1 client.25171 shutdown 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.718+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 msgr2=0x558874074050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.718+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x558874074050 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55887407db40 tx=0x7f77bc010e50 comp rx=0 tx=0).stop 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 shutdown_connections 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5588740eaca0 0x5588740ed0c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55887406a9b0 0x558874075530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558874065840 0x558874074590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 --2- 172.21.15.16:0/2992197526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558874064e50 0x558874074050 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 >> 172.21.15.16:0/2992197526 conn(0x5588740604c0 msgr2=0x55887405ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77aaffd640 0 client.25171 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 shutdown_connections 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 1 -- 172.21.15.16:0/2992197526 wait complete. 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.719+0000 7f77d49b2740 20 client.25171 trim_cache size 0 max 16384 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:30:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:30:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f77b40a2da0 msgr2=0x7f77b40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f77b40a2da0 0x7f77b40a5260 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x5588740660a0 tx=0x7f77c4010040 comp rx=0 tx=0).stop 2024-04-25T01:30:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1018b0 msgr2=0x7f77cc19a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1018b0 0x7f77cc19a070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f77c002a860 tx=0x7f77c0030e00 comp rx=0 tx=0).stop 2024-04-25T01:30:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 shutdown_connections 2024-04-25T01:30:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f77b40a2da0 0x7f77b40a5260 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc102270 0x7f77cc1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77cc1018b0 0x7f77cc19a070 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 --2- 172.21.15.16:0/2644155702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77cc1006c0 0x7f77cc199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 >> 172.21.15.16:0/2644155702 conn(0x7f77cc0fbc30 msgr2=0x7f77cc10a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 shutdown_connections 2024-04-25T01:30:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:26.720+0000 7f77d1bc8640 1 -- 172.21.15.16:0/2644155702 wait complete. 2024-04-25T01:30:27.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:27 smithi016 ceph-mon[29153]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-25T01:30:27.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:27 smithi038 ceph-mon[32841]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-25T01:30:27.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:27 smithi156 ceph-mon[32631]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-25T01:30:29.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:29 smithi016 ceph-mon[29153]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:30:29.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:29 smithi038 ceph-mon[32841]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:30:29.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:29 smithi156 ceph-mon[32631]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:30:31.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:31 smithi016 ceph-mon[29153]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T01:30:31.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:31 smithi038 ceph-mon[32841]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T01:30:31.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:31 smithi156 ceph-mon[32631]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T01:30:33.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:33 smithi016 ceph-mon[29153]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:30:33.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:33 smithi038 ceph-mon[32841]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:30:33.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:33 smithi156 ceph-mon[32631]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:30:35.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:35 smithi016 ceph-mon[29153]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T01:30:35.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:35 smithi038 ceph-mon[32841]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T01:30:35.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:35 smithi156 ceph-mon[32631]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T01:30:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:37 smithi016 ceph-mon[29153]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:30:37.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:37 smithi038 ceph-mon[32841]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:30:37.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:37 smithi156 ceph-mon[32631]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 -- 172.21.15.16:0/211333430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 msgr2=0x7f8520110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 --2- 172.21.15.16:0/211333430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 0x7f8520110210 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8518009ca0 tx=0x7f851802a8b0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 -- 172.21.15.16:0/211333430 shutdown_connections 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 --2- 172.21.15.16:0/211333430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 0x7f8520110210 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 --2- 172.21.15.16:0/211333430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 0x7f8520109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 --2- 172.21.15.16:0/211333430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f85201081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 -- 172.21.15.16:0/211333430 >> 172.21.15.16:0/211333430 conn(0x7f8520076d90 msgr2=0x7f85200791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 -- 172.21.15.16:0/211333430 shutdown_connections 2024-04-25T01:30:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.871+0000 7f8526015640 1 -- 172.21.15.16:0/211333430 wait complete. 2024-04-25T01:30:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.872+0000 7f8526015640 1 Processor -- start 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.872+0000 7f8526015640 1 -- start start 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.872+0000 7f8526015640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f852019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f8526015640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 0x7f852019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f8526015640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 0x7f85201a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f8526015640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8520113930 con 0x7f8520108fc0 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f8526015640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f85201137b0 con 0x7f8520109980 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f8526015640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8520113ab0 con 0x7f8520107dd0 2024-04-25T01:30:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f852019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f852019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:55282/0 (socket says 172.21.15.16:55282) 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 -- 172.21.15.16:0/2001668593 learned_addr learned my addr 172.21.15.16:0/2001668593 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851ffff640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 0x7f85201a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851effd640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 0x7f852019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 -- 172.21.15.16:0/2001668593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 msgr2=0x7f85201a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 0x7f85201a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 -- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 msgr2=0x7f852019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 0x7f852019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851f7fe640 1 -- 172.21.15.16:0/2001668593 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85201a5b00 con 0x7f8520107dd0 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.873+0000 7f851effd640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 0x7f852019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.874+0000 7f851f7fe640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f852019dee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f851000e9c0 tx=0x7f851000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.874+0000 7f851cff9640 1 -- 172.21.15.16:0/2001668593 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851000cd80 con 0x7f8520107dd0 2024-04-25T01:30:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.874+0000 7f851cff9640 1 -- 172.21.15.16:0/2001668593 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85100035f0 con 0x7f8520107dd0 2024-04-25T01:30:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.874+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85201a5df0 con 0x7f8520107dd0 2024-04-25T01:30:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.874+0000 7f851cff9640 1 -- 172.21.15.16:0/2001668593 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510010690 con 0x7f8520107dd0 2024-04-25T01:30:39.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.874+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85201a62b0 con 0x7f8520107dd0 2024-04-25T01:30:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 Processor -- start 2024-04-25T01:30:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f851cff9640 1 -- 172.21.15.16:0/2001668593 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f85100037a0 con 0x7f8520107dd0 2024-04-25T01:30:39.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 -- start start 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa3e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa43340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa45e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56365f9a1cb0 con 0x56365fa3e840 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56365f94a5b0 con 0x56365fa3de50 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f8528e03740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56365f99dc70 con 0x56365fa439b0 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f851effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa45e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f851effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa45e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:55288/0 (socket says 172.21.15.16:55288) 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f851effd640 1 -- 172.21.15.16:0/110075550 learned_addr learned my addr 172.21.15.16:0/110075550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.878+0000 7f851ffff640 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa3e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851f7fe640 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa43340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851effd640 1 -- 172.21.15.16:0/110075550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 msgr2=0x56365fa3e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851effd640 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa3e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851effd640 1 -- 172.21.15.16:0/110075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 msgr2=0x56365fa43340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851effd640 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa43340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851effd640 1 -- 172.21.15.16:0/110075550 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56365f6b6210 con 0x56365fa439b0 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851effd640 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa45e40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8514009cf0 tx=0x7f85140117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca8d61a4e5d908f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f8505ffb640 1 -- 172.21.15.16:0/110075550 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514012450 con 0x56365fa439b0 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f851cff9640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85080a2cb0 0x7f85080a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f8505ffb640 1 -- 172.21.15.16:0/110075550 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8514012aa0 con 0x56365fa439b0 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.879+0000 7f8505ffb640 1 -- 172.21.15.16:0/110075550 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851400fb80 con 0x56365fa439b0 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.880+0000 7f851ffff640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85080a2cb0 0x7f85080a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.880+0000 7f851cff9640 1 -- 172.21.15.16:0/2001668593 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f85100cf4a0 con 0x7f8520107dd0 2024-04-25T01:30:39.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:39 smithi016 ceph-mon[29153]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.880+0000 7f8528e03740 1 -- 172.21.15.16:0/110075550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 msgr2=0x56365fa45e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.880+0000 7f8528e03740 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa45e40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8514009cf0 tx=0x7f85140117b0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.880+0000 7f851ffff640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85080a2cb0 0x7f85080a5170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x56365fa3f0a0 tx=0x7f8518030860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 -- 172.21.15.16:0/110075550 shutdown_connections 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa45e40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa43340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 --2- 172.21.15.16:0/110075550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa3e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 -- 172.21.15.16:0/110075550 >> 172.21.15.16:0/110075550 conn(0x56365fa394c0 msgr2=0x56365fa398c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 -- 172.21.15.16:0/110075550 shutdown_connections 2024-04-25T01:30:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 -- 172.21.15.16:0/110075550 wait complete. 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.881+0000 7f8528e03740 1 -- start start 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.882+0000 7f8528e03740 1 Processor -- start 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.882+0000 7f8528e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa48320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.882+0000 7f8528e03740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3e840 0x56365fa48860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa48320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa48320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58298/0 (socket says 172.21.15.16:58298) 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f8528e03740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa4f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851f7fe640 1 -- 172.21.15.16:0/3437426187 learned_addr learned my addr 172.21.15.16:0/3437426187 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56365fa46960 con 0x56365fa3de50 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56365f95db50 con 0x56365fa3e840 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56365fa46ae0 con 0x56365fa439b0 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851effd640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3e840 0x56365fa48860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851ffff640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa4f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851effd640 1 -- 172.21.15.16:0/3437426187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 msgr2=0x56365fa4f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851effd640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa4f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851effd640 1 -- 172.21.15.16:0/3437426187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 msgr2=0x56365fa48320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851effd640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa48320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851effd640 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56365f782620 con 0x56365fa3e840 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.883+0000 7f851f7fe640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa48320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f8528e03740 10 client.0 _ll_register_callbacks cb 0x56365f998ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f851effd640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3e840 0x56365fa48860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8514000c00 tx=0x7f851400f4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1f4d6fbbacce32b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f84feffd640 10 client.25208 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8514012aa0 con 0x56365fa3e840 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f84feffd640 1 -- 172.21.15.16:0/3437426187 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514029020 con 0x56365fa3e840 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f84feffd640 1 -- 172.21.15.16:0/3437426187 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8514004250 con 0x56365fa3e840 2024-04-25T01:30:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f84feffd640 1 -- 172.21.15.16:0/3437426187 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f85140044d0 con 0x56365fa3e840 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f8528e03740 10 client.25208 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.884+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=22}) -- 0x56365fa52a70 con 0x56365fa3e840 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f84feffd640 1 -- 172.21.15.16:0/3437426187 <== mon.2 v2:172.21.15.38:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8514012450 con 0x56365fa3e840 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f8528e03740 10 client.25208 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f8528e03740 10 client.25208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f8528e03740 20 client.25208 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f8528e03740 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fa561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f8528e03740 4 client.25208 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8514012aa0 con 0x56365fa53dc0 2024-04-25T01:30:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.885+0000 7f851f7fe640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fa561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.886+0000 7f851f7fe640 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fa561b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec5c25f97a4c4c0 server_cookie=80dfa911342ce1a6 in_seq=0 out_seq=0 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.886+0000 7f84feffd640 10 client.25208 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.887+0000 7f84feffd640 1 -- 172.21.15.16:0/3437426187 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8514012aa0 con 0x56365fa53dc0 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.887+0000 7f84feffd640 10 client.25208 handle_command_reply: tid=0 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.887+0000 7f8528e03740 1 client.25208 shutdown 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.887+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3e840 msgr2=0x56365fa48860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3e840 0x56365fa48860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8514000c00 tx=0x7f851400f4b0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 shutdown_connections 2024-04-25T01:30:39.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fa4f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fa561b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f84feffd640 0 client.25208 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3e840 0x56365fa48860 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 --2- 172.21.15.16:0/3437426187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa48320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 >> 172.21.15.16:0/3437426187 conn(0x56365fa394c0 msgr2=0x56365fa385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 shutdown_connections 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 1 -- 172.21.15.16:0/3437426187 wait complete. 2024-04-25T01:30:39.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.888+0000 7f8528e03740 20 client.25208 trim_cache size 0 max 16384 2024-04-25T01:30:39.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.914+0000 7f8528e03740 1 Processor -- start 2024-04-25T01:30:39.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.914+0000 7f8528e03740 1 -- start start 2024-04-25T01:30:39.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f8528e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa3fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f8528e03740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa3e840 0x56365fa4f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f8528e03740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa439b0 0x56365fa40280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa3fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f8528e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85201b19d0 con 0x56365fa3de50 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f8528e03740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f85201af2a0 con 0x56365fa439b0 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa3fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58302/0 (socket says 172.21.15.16:58302) 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f8528e03740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85201ad0f0 con 0x56365fa3e840 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 -- 172.21.15.16:0/2182160363 learned_addr learned my addr 172.21.15.16:0/2182160363 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851ffff640 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa439b0 0x56365fa40280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851effd640 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa3e840 0x56365fa4f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 -- 172.21.15.16:0/2182160363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa3e840 msgr2=0x56365fa4f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa3e840 0x56365fa4f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 -- 172.21.15.16:0/2182160363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa439b0 msgr2=0x56365fa40280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa439b0 0x56365fa40280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.915+0000 7f851f7fe640 1 -- 172.21.15.16:0/2182160363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56365fa3d450 con 0x56365fa3de50 2024-04-25T01:30:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.916+0000 7f851f7fe640 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa3fd40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f851009b980 tx=0x7f851009ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a0eaffb8c76370 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.916+0000 7f85057fa640 1 -- 172.21.15.16:0/2182160363 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510003b30 con 0x56365fa3de50 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.916+0000 7f85057fa640 1 -- 172.21.15.16:0/2182160363 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f851009c380 con 0x56365fa3de50 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.916+0000 7f85057fa640 1 -- 172.21.15.16:0/2182160363 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85100d2d30 con 0x56365fa3de50 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 -- 172.21.15.16:0/2182160363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 msgr2=0x56365fa3fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa3fd40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f851009b980 tx=0x7f851009ba50 comp rx=0 tx=0).stop 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 -- 172.21.15.16:0/2182160363 shutdown_connections 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa439b0 0x56365fa40280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa3e840 0x56365fa4f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 --2- 172.21.15.16:0/2182160363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3de50 0x56365fa3fd40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 -- 172.21.15.16:0/2182160363 >> 172.21.15.16:0/2182160363 conn(0x56365fa394c0 msgr2=0x56365fa398a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 -- 172.21.15.16:0/2182160363 shutdown_connections 2024-04-25T01:30:39.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 -- 172.21.15.16:0/2182160363 wait complete. 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 -- start start 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.917+0000 7f8528e03740 1 Processor -- start 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa4c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa4cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fab8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f85201ad760 con 0x56365fa3e840 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f85201aebd0 con 0x56365fa3de50 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85201ade30 con 0x56365fa439b0 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa4cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa4cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58310/0 (socket says 172.21.15.16:58310) 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 -- 172.21.15.16:0/1405818001 learned_addr learned my addr 172.21.15.16:0/1405818001 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851f7fe640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa4c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.899 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851ffff640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fab8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f8528e03740 10 client.0 _ll_register_callbacks cb 0x7f8510005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 -- 172.21.15.16:0/1405818001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 msgr2=0x56365fab8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fab8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 -- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 msgr2=0x56365fa4c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa4c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851effd640 1 -- 172.21.15.16:0/1405818001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56365fab87f0 con 0x56365fa3e840 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.918+0000 7f851f7fe640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa4c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f851effd640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa4cdf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x56365fa49800 tx=0x7f8514005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5871896eaa26e868 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f84feffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f84feffd640 1 -- 172.21.15.16:0/1405818001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514009b40 con 0x56365fa3e840 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f84feffd640 1 -- 172.21.15.16:0/1405818001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f851400f680 con 0x56365fa3e840 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f851009c380 con 0x56365fa3e840 2024-04-25T01:30:39.900 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f84feffd640 1 -- 172.21.15.16:0/1405818001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8514048050 con 0x56365fa3e840 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f8528e03740 10 client.15540 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56365fab9ae0 con 0x56365fa3e840 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.919+0000 7f84feffd640 1 -- 172.21.15.16:0/1405818001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f851009c380 con 0x56365fa3e840 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f84feffd640 1 -- 172.21.15.16:0/1405818001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8514003780 con 0x56365fa3e840 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f8528e03740 10 client.15540 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f8528e03740 10 client.15540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f8528e03740 20 client.15540 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f8528e03740 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fabc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f8528e03740 4 client.15540 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56365f9af1f0 con 0x56365fa53dc0 2024-04-25T01:30:39.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f851f7fe640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fabc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.920+0000 7f851f7fe640 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fabc4a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b631a7c393365116 server_cookie=f23844f3cf909520 in_seq=0 out_seq=0 2024-04-25T01:30:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.921+0000 7f84feffd640 10 client.15540 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:39.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.925+0000 7f84feffd640 1 -- 172.21.15.16:0/1405818001 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56365f9af1f0 con 0x56365fa53dc0 2024-04-25T01:30:39.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.925+0000 7f84feffd640 10 client.15540 handle_command_reply: tid=0 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.925+0000 7f8528e03740 1 client.15540 shutdown 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.925+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 msgr2=0x56365fa4cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.925+0000 7f8528e03740 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa4cdf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x56365fa49800 tx=0x7f8514005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 shutdown_connections 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56365fa439b0 0x56365fab8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56365fa3e840 0x56365fa4cdf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56365fa53dc0 0x56365fabc4a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 --2- 172.21.15.16:0/1405818001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56365fa3de50 0x56365fa4c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 >> 172.21.15.16:0/1405818001 conn(0x56365fa394c0 msgr2=0x56365fa3fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f84feffd640 0 client.15540 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 shutdown_connections 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 1 -- 172.21.15.16:0/1405818001 wait complete. 2024-04-25T01:30:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.926+0000 7f8528e03740 20 client.15540 trim_cache size 0 max 16384 2024-04-25T01:30:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:30:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:30:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:30:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85080a2cb0 msgr2=0x7f85080a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85080a2cb0 0x7f85080a5170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x56365fa3f0a0 tx=0x7f8518030860 comp rx=0 tx=0).stop 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 msgr2=0x7f852019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f852019dee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f851000e9c0 tx=0x7f851000ee80 comp rx=0 tx=0).stop 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 shutdown_connections 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f85080a2cb0 0x7f85080a5170 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8520109980 0x7f85201a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8520108fc0 0x7f852019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 --2- 172.21.15.16:0/2001668593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8520107dd0 0x7f852019dee0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 >> 172.21.15.16:0/2001668593 conn(0x7f8520076d90 msgr2=0x7f852010c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 shutdown_connections 2024-04-25T01:30:39.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:39.927+0000 7f8526015640 1 -- 172.21.15.16:0/2001668593 wait complete. 2024-04-25T01:30:39.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:39 smithi038 ceph-mon[32841]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-04-25T01:30:39.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:39 smithi156 ceph-mon[32631]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-04-25T01:30:41.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:41 smithi016 ceph-mon[29153]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:30:41.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:41 smithi038 ceph-mon[32841]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:30:41.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:41 smithi156 ceph-mon[32631]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:30:43.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:43 smithi016 ceph-mon[29153]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:30:43.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:43 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:43.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:43 smithi038 ceph-mon[32841]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:30:43.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:43 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:43.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:43 smithi156 ceph-mon[32631]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:30:43.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:43 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:44.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:44 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:44.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:44 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:44.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:44 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:30:44.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:44 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:44.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:44 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:44.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:44 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:30:44.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:44 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:44.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:44 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:44.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:44 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:30:45.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:45 smithi016 ceph-mon[29153]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T01:30:45.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:45 smithi038 ceph-mon[32841]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T01:30:45.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:45 smithi156 ceph-mon[32631]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T01:30:46.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:46 smithi016 ceph-mon[29153]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:30:46.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:46 smithi038 ceph-mon[32841]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:30:46.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:46 smithi156 ceph-mon[32631]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:30:48.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:48 smithi156 ceph-mon[32631]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:30:49.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:48 smithi016 ceph-mon[29153]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:30:49.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:48 smithi038 ceph-mon[32841]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:30:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:50 smithi156 ceph-mon[32631]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:30:51.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:50 smithi016 ceph-mon[29153]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:30:51.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:50 smithi038 ceph-mon[32841]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:30:52.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:52 smithi156 ceph-mon[32631]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:30:53.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f2008adc640 1 -- 172.21.15.16:0/203950182 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4031880 con 0x7f2004104bc0 2024-04-25T01:30:53.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 -- 172.21.15.16:0/203950182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2004104bc0 msgr2=0x7f2004105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 --2- 172.21.15.16:0/203950182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2004104bc0 0x7f2004105040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009900 tx=0x7f1ff402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:53.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 -- 172.21.15.16:0/203950182 shutdown_connections 2024-04-25T01:30:53.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 --2- 172.21.15.16:0/203950182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2004105580 0x7f200410be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 --2- 172.21.15.16:0/203950182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2004104bc0 0x7f2004105040 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 --2- 172.21.15.16:0/203950182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 0x7f2004103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.075+0000 7f200bd42640 1 -- 172.21.15.16:0/203950182 >> 172.21.15.16:0/203950182 conn(0x7f20040fef60 msgr2=0x7f2004101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:53.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.076+0000 7f200bd42640 1 -- 172.21.15.16:0/203950182 shutdown_connections 2024-04-25T01:30:53.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.076+0000 7f200bd42640 1 -- 172.21.15.16:0/203950182 wait complete. 2024-04-25T01:30:53.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 Processor -- start 2024-04-25T01:30:53.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 -- start start 2024-04-25T01:30:53.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 0x7f2004199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2004105580 0x7f200419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 0x7f20041a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f200410f570 con 0x7f2004105580 2024-04-25T01:30:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f200410f3f0 con 0x7f20041039d0 2024-04-25T01:30:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.077+0000 7f200bd42640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f200410f6f0 con 0x7f200419f3c0 2024-04-25T01:30:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f2009ade640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 0x7f2004199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f2009ade640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 0x7f2004199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:52084/0 (socket says 172.21.15.16:52084) 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f200a2df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 0x7f20041a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f200a2df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 0x7f20041a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34480/0 (socket says 172.21.15.16:34480) 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f200a2df640 1 -- 172.21.15.16:0/3682998242 learned_addr learned my addr 172.21.15.16:0/3682998242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f20092dd640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2004105580 0x7f200419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f200a2df640 1 -- 172.21.15.16:0/3682998242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 msgr2=0x7f2004199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.078+0000 7f200a2df640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 0x7f2004199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f200a2df640 1 -- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2004105580 msgr2=0x7f200419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f200a2df640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2004105580 0x7f200419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f200a2df640 1 -- 172.21.15.16:0/3682998242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200406b290 con 0x7f200419f3c0 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f200a2df640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 0x7f20041a17b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00efd0 tx=0x7f1ffc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f1ffaffd640 1 -- 172.21.15.16:0/3682998242 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc003b70 con 0x7f200419f3c0 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200406b580 con 0x7f200419f3c0 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200406bb20 con 0x7f200419f3c0 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f1ffaffd640 1 -- 172.21.15.16:0/3682998242 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ffc003d10 con 0x7f200419f3c0 2024-04-25T01:30:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.079+0000 7f1ffaffd640 1 -- 172.21.15.16:0/3682998242 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc010980 con 0x7f200419f3c0 2024-04-25T01:30:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.083+0000 7f1ffaffd640 1 -- 172.21.15.16:0/3682998242 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1ffc010b20 con 0x7f200419f3c0 2024-04-25T01:30:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.085+0000 7f1ffaffd640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ff00a2cb0 0x7f1ff00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.085+0000 7f200ea50740 1 Processor -- start 2024-04-25T01:30:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.085+0000 7f200ea50740 1 -- start start 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f200ea50740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85413270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f200ea50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed85418340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f200ea50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed8541ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f200ea50740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ed85376cb0 con 0x55ed85412e50 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f200ea50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ed8531f5b0 con 0x55ed854189b0 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f200ea50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ed85372c70 con 0x55ed85413840 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f2009ade640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ff00a2cb0 0x7f1ff00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f20092dd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed85418340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f20092dd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed85418340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34488/0 (socket says 172.21.15.16:34488) 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.086+0000 7f1ffaffd640 1 -- 172.21.15.16:0/3682998242 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1ffc0cf370 con 0x7f200419f3c0 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.087+0000 7f20092dd640 1 -- 172.21.15.16:0/1463791408 learned_addr learned my addr 172.21.15.16:0/1463791408 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.087+0000 7f200a2df640 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed8541ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.087+0000 7f2009ade640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ff00a2cb0 0x7f1ff00a5170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2004104a20 tx=0x7f20000090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f200a2df640 1 -- 172.21.15.16:0/1463791408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 msgr2=0x55ed85418340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f200a2df640 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed85418340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f200a2df640 1 -- 172.21.15.16:0/1463791408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 msgr2=0x55ed85413270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f200a2df640 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85413270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f200a2df640 1 -- 172.21.15.16:0/1463791408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed8508b210 con 0x55ed854189b0 2024-04-25T01:30:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f20092dd640 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed85418340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:53.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f200a2df640 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed8541ae40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00cba0 tx=0x7f1ffc09b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7509ee81e9ed37e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:53.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.088+0000 7f1feffff640 1 -- 172.21.15.16:0/1463791408 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0b3070 con 0x55ed854189b0 2024-04-25T01:30:53.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.089+0000 7f1feffff640 1 -- 172.21.15.16:0/1463791408 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ffc09ff00 con 0x55ed854189b0 2024-04-25T01:30:53.067 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.089+0000 7f1feffff640 1 -- 172.21.15.16:0/1463791408 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc097bd0 con 0x55ed854189b0 2024-04-25T01:30:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 -- 172.21.15.16:0/1463791408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 msgr2=0x55ed8541ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.068 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed8541ae40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00cba0 tx=0x7f1ffc09b5d0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 -- 172.21.15.16:0/1463791408 shutdown_connections 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed8541ae40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed85418340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 --2- 172.21.15.16:0/1463791408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85413270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 -- 172.21.15.16:0/1463791408 >> 172.21.15.16:0/1463791408 conn(0x55ed8540e4c0 msgr2=0x55ed8540e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 -- 172.21.15.16:0/1463791408 shutdown_connections 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.090+0000 7f200ea50740 1 -- 172.21.15.16:0/1463791408 wait complete. 2024-04-25T01:30:53.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.091+0000 7f200ea50740 1 -- start start 2024-04-25T01:30:53.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.091+0000 7f200ea50740 1 Processor -- start 2024-04-25T01:30:53.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85425a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85413840 0x55ed85425f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8542cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ed85424150 con 0x55ed85412e50 2024-04-25T01:30:53.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ed85332b50 con 0x55ed85413840 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ed854242d0 con 0x55ed854189b0 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200ea50740 10 client.0 _ll_register_callbacks cb 0x55ed8536dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200a2df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8542cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200a2df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8542cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34496/0 (socket says 172.21.15.16:34496) 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f200a2df640 1 -- 172.21.15.16:0/384236551 learned_addr learned my addr 172.21.15.16:0/384236551 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f20092dd640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85413840 0x55ed85425f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.092+0000 7f2009ade640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85425a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.093+0000 7f20092dd640 1 -- 172.21.15.16:0/384236551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 msgr2=0x55ed8542cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.093+0000 7f20092dd640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8542cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.093+0000 7f20092dd640 1 -- 172.21.15.16:0/384236551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 msgr2=0x55ed85425a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.093+0000 7f20092dd640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85425a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.093+0000 7f20092dd640 1 -- 172.21.15.16:0/384236551 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed85157620 con 0x55ed85413840 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.093+0000 7f200a2df640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8542cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.094+0000 7f20092dd640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85413840 0x55ed85425f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55ed854142b0 tx=0x7f1ff4031580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1caca71023661b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.094+0000 7f1fed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.094+0000 7f1fed7fa640 1 -- 172.21.15.16:0/384236551 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4035040 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.094+0000 7f1fed7fa640 1 -- 172.21.15.16:0/384236551 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1ff4004040 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.095+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ffc09ff00 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.095+0000 7f1fed7fa640 1 -- 172.21.15.16:0/384236551 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1ff4004240 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.095+0000 7f200ea50740 10 client.25223 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.095+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ed854303a0 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.095+0000 7f1fed7fa640 1 -- 172.21.15.16:0/384236551 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ffc09ff00 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.095+0000 7f1fed7fa640 1 -- 172.21.15.16:0/384236551 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1ff403d450 con 0x55ed85413840 2024-04-25T01:30:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f200ea50740 10 client.25223 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f200ea50740 10 client.25223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f200ea50740 20 client.25223 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f200ea50740 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85431a80 0x55ed85433ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f200ea50740 4 client.25223 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed853841f0 con 0x55ed85431a80 2024-04-25T01:30:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.096+0000 7f2009ade640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85431a80 0x55ed85433ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.097+0000 7f2009ade640 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85431a80 0x55ed85433ed0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=309c0ec7e8ad52f9 server_cookie=f56a60bb6043b692 in_seq=0 out_seq=0 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.097+0000 7f1fed7fa640 10 client.25223 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f1fed7fa640 1 -- 172.21.15.16:0/384236551 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed853841f0 con 0x55ed85431a80 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f1fed7fa640 10 client.25223 handle_command_reply: tid=0 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 client.25223 shutdown 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85413840 msgr2=0x55ed85425f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85413840 0x55ed85425f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55ed854142b0 tx=0x7f1ff4031580 comp rx=0 tx=0).stop 2024-04-25T01:30:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 shutdown_connections 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8542cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85431a80 0x55ed85433ed0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85413840 0x55ed85425f80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 --2- 172.21.15.16:0/384236551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed85425a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 >> 172.21.15.16:0/384236551 conn(0x55ed8540e4c0 msgr2=0x55ed8540d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f1fed7fa640 0 client.25223 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.098+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 shutdown_connections 2024-04-25T01:30:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.099+0000 7f200ea50740 1 -- 172.21.15.16:0/384236551 wait complete. 2024-04-25T01:30:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.099+0000 7f200ea50740 20 client.25223 trim_cache size 0 max 16384 2024-04-25T01:30:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.128+0000 7f200ea50740 1 Processor -- start 2024-04-25T01:30:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.128+0000 7f200ea50740 1 -- start start 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200ea50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 0x55ed85418270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200ea50740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85413840 0x55ed8542cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200ea50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8540e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200ea50740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe8010df0 con 0x55ed85413840 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200ea50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe800e6c0 con 0x55ed85412e50 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200ea50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe800c510 con 0x55ed854189b0 2024-04-25T01:30:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f20092dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85413840 0x55ed8542cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f2009ade640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 0x55ed85418270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f2009ade640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 0x55ed85418270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:52106/0 (socket says 172.21.15.16:52106) 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f2009ade640 1 -- 172.21.15.16:0/4282393638 learned_addr learned my addr 172.21.15.16:0/4282393638 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.129+0000 7f200a2df640 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8540e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f2009ade640 1 -- 172.21.15.16:0/4282393638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 msgr2=0x55ed8540e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f2009ade640 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8540e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f2009ade640 1 -- 172.21.15.16:0/4282393638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85413840 msgr2=0x55ed8542cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f2009ade640 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85413840 0x55ed8542cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f2009ade640 1 -- 172.21.15.16:0/4282393638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed8540e060 con 0x55ed85412e50 2024-04-25T01:30:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f20092dd640 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85413840 0x55ed8542cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f2009ade640 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 0x55ed85418270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f200000e5a0 tx=0x7f2000011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ddd74a6b4b6f6a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:53.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:52 smithi016 ceph-mon[29153]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:30:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f1fef7fe640 1 -- 172.21.15.16:0/4282393638 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200000ee20 con 0x55ed85412e50 2024-04-25T01:30:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f1fef7fe640 1 -- 172.21.15.16:0/4282393638 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f200002a030 con 0x55ed85412e50 2024-04-25T01:30:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.130+0000 7f1fef7fe640 1 -- 172.21.15.16:0/4282393638 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000013a90 con 0x55ed85412e50 2024-04-25T01:30:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.131+0000 7f200ea50740 1 -- 172.21.15.16:0/4282393638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 msgr2=0x55ed85418270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.131+0000 7f200ea50740 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 0x55ed85418270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f200000e5a0 tx=0x7f2000011000 comp rx=0 tx=0).stop 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 -- 172.21.15.16:0/4282393638 shutdown_connections 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed854189b0 0x55ed8540e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85413840 0x55ed8542cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 --2- 172.21.15.16:0/4282393638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed85412e50 0x55ed85418270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 -- 172.21.15.16:0/4282393638 >> 172.21.15.16:0/4282393638 conn(0x55ed8540e4c0 msgr2=0x55ed854092d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 -- 172.21.15.16:0/4282393638 shutdown_connections 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 -- 172.21.15.16:0/4282393638 wait complete. 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 -- start start 2024-04-25T01:30:53.111 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.132+0000 7f200ea50740 1 Processor -- start 2024-04-25T01:30:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed8542cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed854299e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f2009ade640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed8542cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f2009ade640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed8542cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58146/0 (socket says 172.21.15.16:58146) 2024-04-25T01:30:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed85429fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe800cb80 con 0x55ed85412e50 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe800dff0 con 0x55ed854189b0 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe800d250 con 0x55ed85413840 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f2009ade640 1 -- 172.21.15.16:0/2781618277 learned_addr learned my addr 172.21.15.16:0/2781618277 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f20092dd640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed854299e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.133+0000 7f200ea50740 10 client.0 _ll_register_callbacks cb 0x7f1ff4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f200a2df640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed85429fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f2009ade640 1 -- 172.21.15.16:0/2781618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 msgr2=0x55ed854299e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f2009ade640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed854299e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f2009ade640 1 -- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 msgr2=0x55ed85429fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f2009ade640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed85429fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f2009ade640 1 -- 172.21.15.16:0/2781618277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed85496070 con 0x55ed85412e50 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f200a2df640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed85429fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f2009ade640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed8542cf40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f200000eab0 tx=0x7f200000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be3aa71f008ef65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f1fed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f1fed7fa640 1 -- 172.21.15.16:0/2781618277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000265f0 con 0x55ed85412e50 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.134+0000 7f1fed7fa640 1 -- 172.21.15.16:0/2781618277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2000026790 con 0x55ed85412e50 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.135+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f200002a030 con 0x55ed85412e50 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.135+0000 7f1fed7fa640 1 -- 172.21.15.16:0/2781618277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2000013790 con 0x55ed85412e50 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.135+0000 7f200ea50740 10 client.15555 fetch_fsmap learned FSMap version 22 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.135+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ed85497360 con 0x55ed85412e50 2024-04-25T01:30:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.135+0000 7f1fed7fa640 1 -- 172.21.15.16:0/2781618277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f200002a030 con 0x55ed85412e50 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f1fed7fa640 1 -- 172.21.15.16:0/2781618277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f2000042020 con 0x55ed85412e50 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f200ea50740 10 client.15555 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f200ea50740 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f200ea50740 20 client.15555 populate_metadata read hostname 'smithi016' 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f200ea50740 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85497c60 0x55ed8549a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f200ea50740 4 client.15555 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ed8537e540 con 0x55ed85497c60 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.136+0000 7f20092dd640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85497c60 0x55ed8549a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.137+0000 7f1fed7fa640 10 client.15555 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:53.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.137+0000 7f20092dd640 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85497c60 0x55ed8549a080 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1279530d6c923f01 server_cookie=76dfeef67404f8ef in_seq=0 out_seq=0 2024-04-25T01:30:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.146+0000 7f1fed7fa640 1 -- 172.21.15.16:0/2781618277 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ed8537e540 con 0x55ed85497c60 2024-04-25T01:30:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.146+0000 7f1fed7fa640 10 client.15555 handle_command_reply: tid=0 2024-04-25T01:30:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.146+0000 7f200ea50740 1 client.15555 shutdown 2024-04-25T01:30:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.146+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 msgr2=0x55ed8542cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.146+0000 7f200ea50740 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed8542cf40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f200000eab0 tx=0x7f200000eb70 comp rx=0 tx=0).stop 2024-04-25T01:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 shutdown_connections 2024-04-25T01:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f200ea50740 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55ed85497c60 0x55ed8549a080 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f200ea50740 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ed854189b0 0x55ed85429fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f200ea50740 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ed85413840 0x55ed854299e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f200ea50740 1 --2- 172.21.15.16:0/2781618277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed85412e50 0x55ed8542cf40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 >> 172.21.15.16:0/2781618277 conn(0x55ed8540e4c0 msgr2=0x55ed854180e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.147+0000 7f1fed7fa640 0 client.15555 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.148+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 shutdown_connections 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.148+0000 7f200ea50740 1 -- 172.21.15.16:0/2781618277 wait complete. 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.148+0000 7f200ea50740 20 client.15555 trim_cache size 0 max 16384 2024-04-25T01:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ff00a2cb0 msgr2=0x7f1ff00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ff00a2cb0 0x7f1ff00a5170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2004104a20 tx=0x7f20000090e0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 msgr2=0x7f20041a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 0x7f20041a17b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00efd0 tx=0x7f1ffc00c6a0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 shutdown_connections 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ff00a2cb0 0x7f1ff00a5170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f200419f3c0 0x7f20041a17b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2004105580 0x7f200419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 --2- 172.21.15.16:0/3682998242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20041039d0 0x7f2004199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 >> 172.21.15.16:0/3682998242 conn(0x7f20040fef60 msgr2=0x7f2004107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 shutdown_connections 2024-04-25T01:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:30:53.149+0000 7f200bd42640 1 -- 172.21.15.16:0/3682998242 wait complete. 2024-04-25T01:30:53.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:52 smithi038 ceph-mon[32841]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:30:54.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:54 smithi156 ceph-mon[32631]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:30:55.060 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:54 smithi038 ceph-mon[32841]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:30:55.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:54 smithi016 ceph-mon[29153]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:30:56.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:56 smithi156 ceph-mon[32631]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:30:57.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:56 smithi016 ceph-mon[29153]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:30:57.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:56 smithi038 ceph-mon[32841]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:30:58.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:30:58 smithi156 ceph-mon[32631]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-04-25T01:30:59.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:30:58 smithi016 ceph-mon[29153]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-04-25T01:30:59.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:30:58 smithi038 ceph-mon[32841]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-04-25T01:31:00.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:00 smithi156 ceph-mon[32631]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:01.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:00 smithi016 ceph-mon[29153]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:01.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:00 smithi038 ceph-mon[32841]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:02.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:02 smithi156 ceph-mon[32631]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:03.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:02 smithi016 ceph-mon[29153]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:03.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:02 smithi038 ceph-mon[32841]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:04.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:04 smithi156 ceph-mon[32631]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:31:05.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:04 smithi016 ceph-mon[29153]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:31:05.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:04 smithi038 ceph-mon[32841]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.294+0000 7f7828ca5640 1 -- 172.21.15.16:0/2565013349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 msgr2=0x7f78241081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.294+0000 7f7828ca5640 1 --2- 172.21.15.16:0/2565013349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f78241081f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7814009900 tx=0x7f781402a410 comp rx=0 tx=0).stop 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 -- 172.21.15.16:0/2565013349 shutdown_connections 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 --2- 172.21.15.16:0/2565013349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 0x7f7824110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 --2- 172.21.15.16:0/2565013349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f7824109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 --2- 172.21.15.16:0/2565013349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f78241081f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 -- 172.21.15.16:0/2565013349 >> 172.21.15.16:0/2565013349 conn(0x7f7824076d90 msgr2=0x7f78240791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 -- 172.21.15.16:0/2565013349 shutdown_connections 2024-04-25T01:31:06.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.295+0000 7f7828ca5640 1 -- 172.21.15.16:0/2565013349 wait complete. 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.296+0000 7f7828ca5640 1 Processor -- start 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.296+0000 7f7828ca5640 1 -- start start 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7828ca5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f782419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7828ca5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f782419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7828ca5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 0x7f78241a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7828ca5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7824113940 con 0x7f7824109980 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7828ca5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f78241137c0 con 0x7f7824108fc0 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7828ca5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7824113ac0 con 0x7f7824107dd0 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f782259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f782419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f782419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f782419e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:58030/0 (socket says 172.21.15.16:58030) 2024-04-25T01:31:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f782259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f782419df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44256/0 (socket says 172.21.15.16:44256) 2024-04-25T01:31:06.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 -- 172.21.15.16:0/1061921153 learned_addr learned my addr 172.21.15.16:0/1061921153 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7822d9d640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 0x7f78241a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 -- 172.21.15.16:0/1061921153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 msgr2=0x7f782419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f782419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 -- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 msgr2=0x7f78241a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 0x7f78241a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f781bfff640 1 -- 172.21.15.16:0/1061921153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78241a5ba0 con 0x7f7824108fc0 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f782259c640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f782419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.297+0000 7f7822d9d640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 0x7f78241a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.298+0000 7f781bfff640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f782419e4c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f780c00e9f0 tx=0x7f780c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.298+0000 7f781b7fe640 1 -- 172.21.15.16:0/1061921153 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c00cd80 con 0x7f7824108fc0 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.298+0000 7f781b7fe640 1 -- 172.21.15.16:0/1061921153 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f780c0054d0 con 0x7f7824108fc0 2024-04-25T01:31:06.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.298+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78241a5e90 con 0x7f7824108fc0 2024-04-25T01:31:06.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.298+0000 7f781b7fe640 1 -- 172.21.15.16:0/1061921153 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c010490 con 0x7f7824108fc0 2024-04-25T01:31:06.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.298+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78241a6300 con 0x7f7824108fc0 2024-04-25T01:31:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.300+0000 7f781b7fe640 1 -- 172.21.15.16:0/1061921153 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f780c005020 con 0x7f7824108fc0 2024-04-25T01:31:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.301+0000 7f781b7fe640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78100a2cb0 0x7f78100a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.301+0000 7f781b7fe640 1 -- 172.21.15.16:0/1061921153 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f780c0cf4f0 con 0x7f7824108fc0 2024-04-25T01:31:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.301+0000 7f782259c640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78100a2cb0 0x7f78100a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.302+0000 7f782b9b1740 1 Processor -- start 2024-04-25T01:31:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.302+0000 7f782b9b1740 1 -- start start 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.302+0000 7f782259c640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78100a2cb0 0x7f78100a5170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7814005ab0 tx=0x7f7814005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 0x555cec7be340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7be9b0 0x555cec7c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782b9b1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555cec71ccb0 con 0x555cec7be9b0 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782b9b1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555cec6c55b0 con 0x555cec7b8e50 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782b9b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555cec718c70 con 0x555cec7b9840 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 0x555cec7be340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f782259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 0x555cec7be340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44264/0 (socket says 172.21.15.16:44264) 2024-04-25T01:31:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 -- 172.21.15.16:0/4184951994 learned_addr learned my addr 172.21.15.16:0/4184951994 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f7822d9d640 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7be9b0 0x555cec7c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 -- 172.21.15.16:0/4184951994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 msgr2=0x555cec7b9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 -- 172.21.15.16:0/4184951994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7be9b0 msgr2=0x555cec7c0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7be9b0 0x555cec7c0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.303+0000 7f781bfff640 1 -- 172.21.15.16:0/4184951994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cec431210 con 0x555cec7b9840 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.304+0000 7f781bfff640 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 0x555cec7be340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f780c0a4410 tx=0x7f780c09b6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb9106e1ee851d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.304+0000 7f7818ff9640 1 -- 172.21.15.16:0/4184951994 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c09c3d0 con 0x555cec7b9840 2024-04-25T01:31:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.304+0000 7f7818ff9640 1 -- 172.21.15.16:0/4184951994 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f780c09bcd0 con 0x555cec7b9840 2024-04-25T01:31:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.304+0000 7f7818ff9640 1 -- 172.21.15.16:0/4184951994 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c098340 con 0x555cec7b9840 2024-04-25T01:31:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 -- 172.21.15.16:0/4184951994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 msgr2=0x555cec7be340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 0x555cec7be340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f780c0a4410 tx=0x7f780c09b6f0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 -- 172.21.15.16:0/4184951994 shutdown_connections 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7be9b0 0x555cec7c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b9840 0x555cec7be340 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 --2- 172.21.15.16:0/4184951994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 -- 172.21.15.16:0/4184951994 >> 172.21.15.16:0/4184951994 conn(0x555cec7b44c0 msgr2=0x555cec7b48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.305+0000 7f782b9b1740 1 -- 172.21.15.16:0/4184951994 shutdown_connections 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.306+0000 7f782b9b1740 1 -- 172.21.15.16:0/4184951994 wait complete. 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.306+0000 7f782b9b1740 1 -- start start 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.306+0000 7f782b9b1740 1 Processor -- start 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.306+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7b9840 0x555cec7c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7be9b0 0x555cec7ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782b9b1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555cec7c1960 con 0x555cec7b9840 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782b9b1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555cec6d8b50 con 0x555cec7be9b0 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782b9b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555cec7c1ae0 con 0x555cec7b8e50 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782b9b1740 10 client.0 _ll_register_callbacks cb 0x555cec713ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7b9840 0x555cec7c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7c32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44274/0 (socket says 172.21.15.16:44274) 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f782259c640 1 -- 172.21.15.16:0/3762980166 learned_addr learned my addr 172.21.15.16:0/3762980166 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f7822d9d640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7be9b0 0x555cec7ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 -- 172.21.15.16:0/3762980166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 msgr2=0x555cec7c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 -- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7be9b0 msgr2=0x555cec7ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7be9b0 0x555cec7ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 -- 172.21.15.16:0/3762980166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cec4fd620 con 0x555cec7b9840 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f7822d9d640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7be9b0 0x555cec7ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f781bfff640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7b9840 0x555cec7c37e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f780c004930 tx=0x7f780c0a46e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3aa49be72f881a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.307+0000 7f7809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.308+0000 7f7809ffb640 1 -- 172.21.15.16:0/3762980166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c0a4940 con 0x555cec7b9840 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.308+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f780c09bcd0 con 0x555cec7b9840 2024-04-25T01:31:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.308+0000 7f7809ffb640 1 -- 172.21.15.16:0/3762980166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f780c099fd0 con 0x555cec7b9840 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.308+0000 7f7809ffb640 1 -- 172.21.15.16:0/3762980166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f780c09c3d0 con 0x555cec7b9840 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 10 client.15567 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555cec7cdb90 con 0x555cec7b9840 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f7809ffb640 1 -- 172.21.15.16:0/3762980166 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f780c0cf0d0 con 0x555cec7b9840 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f7809ffb640 1 -- 172.21.15.16:0/3762980166 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f780c0b3020 con 0x555cec7b9840 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 10 client.15567 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 20 client.15567 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec7ceee0 0x555cec7d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 4 client.15567 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.309+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f780c09bcd0 con 0x555cec7ceee0 2024-04-25T01:31:06.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.310+0000 7f782259c640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec7ceee0 0x555cec7d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.310+0000 7f782259c640 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec7ceee0 0x555cec7d12d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6dc1dc9dee17117 server_cookie=63264fa143d6646 in_seq=0 out_seq=0 2024-04-25T01:31:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.310+0000 7f7809ffb640 10 client.15567 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:06.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f7809ffb640 1 -- 172.21.15.16:0/3762980166 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f780c09bcd0 con 0x555cec7ceee0 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f7809ffb640 10 client.15567 handle_command_reply: tid=0 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 client.15567 shutdown 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7b9840 msgr2=0x555cec7c37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7b9840 0x555cec7c37e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f780c004930 tx=0x7f780c0a46e0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 shutdown_connections 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec7ceee0 0x555cec7d12d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7be9b0 0x555cec7ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7b9840 0x555cec7c37e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f7809ffb640 0 client.15567 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3762980166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 >> 172.21.15.16:0/3762980166 conn(0x555cec7b44c0 msgr2=0x555cec7b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.311+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 shutdown_connections 2024-04-25T01:31:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.312+0000 7f782b9b1740 1 -- 172.21.15.16:0/3762980166 wait complete. 2024-04-25T01:31:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.312+0000 7f782b9b1740 20 client.15567 trim_cache size 0 max 16384 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 Processor -- start 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 -- start start 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 0x555cec7ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 0x555cec7b3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78241b1ae0 con 0x555cec7bb370 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f78241af3b0 con 0x555cec7b8e50 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782b9b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f78241ad200 con 0x555cec7be9b0 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f7822d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 0x555cec7b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 0x555cec7ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f7822d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 0x555cec7b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51366/0 (socket says 172.21.15.16:51366) 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 0x555cec7ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44290/0 (socket says 172.21.15.16:44290) 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f7822d9d640 1 -- 172.21.15.16:0/3600655243 learned_addr learned my addr 172.21.15.16:0/3600655243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f782259c640 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 -- 172.21.15.16:0/3600655243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 msgr2=0x555cec7b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 -- 172.21.15.16:0/3600655243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 msgr2=0x555cec7b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 0x555cec7b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f781bfff640 1 -- 172.21.15.16:0/3600655243 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cec7bdf50 con 0x555cec7be9b0 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.338+0000 7f7822d9d640 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 0x555cec7b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.339+0000 7f781bfff640 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 0x555cec7ca7e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f780c009000 tx=0x7f780c004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60a4ca76433e937c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.339+0000 7f780bfff640 1 -- 172.21.15.16:0/3600655243 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c0a4940 con 0x555cec7be9b0 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.339+0000 7f780bfff640 1 -- 172.21.15.16:0/3600655243 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f780c09bcd0 con 0x555cec7be9b0 2024-04-25T01:31:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.339+0000 7f780bfff640 1 -- 172.21.15.16:0/3600655243 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c0cf0d0 con 0x555cec7be9b0 2024-04-25T01:31:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 -- 172.21.15.16:0/3600655243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 msgr2=0x555cec7ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 0x555cec7ca7e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f780c009000 tx=0x7f780c004ac0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 -- 172.21.15.16:0/3600655243 shutdown_connections 2024-04-25T01:31:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7bb370 0x555cec7b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7be9b0 0x555cec7ca7e0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f780c009000 tx=0x7f780c004ac0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.321 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3600655243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7b8e50 0x555cec7b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.340+0000 7f782b9b1740 1 -- 172.21.15.16:0/3600655243 >> 172.21.15.16:0/3600655243 conn(0x555cec7b44c0 msgr2=0x555cec7bac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.341+0000 7f782b9b1740 1 -- 172.21.15.16:0/3600655243 shutdown_connections 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.341+0000 7f782b9b1740 1 -- 172.21.15.16:0/3600655243 wait complete. 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.342+0000 7f782b9b1740 1 -- start start 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.342+0000 7f782b9b1740 1 Processor -- start 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.342+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.342+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7bb370 0x555cec7c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f781bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7bb370 0x555cec7c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44294/0 (socket says 172.21.15.16:44294) 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f781bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7bb370 0x555cec7c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:58072/0 (socket says 172.21.15.16:58072) 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782b9b1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7c8fa0 0x555cec7c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782b9b1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78241ad870 con 0x555cec7c8fa0 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782b9b1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f78241aece0 con 0x555cec7bb370 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782b9b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f78241adf40 con 0x555cec7b8e50 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782b9b1740 10 client.0 _ll_register_callbacks cb 0x7f7814040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 -- 172.21.15.16:0/3626223961 learned_addr learned my addr 172.21.15.16:0/3626223961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f7822d9d640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7c8fa0 0x555cec7c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 -- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7bb370 msgr2=0x555cec7c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7bb370 0x555cec7c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 -- 172.21.15.16:0/3626223961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7c8fa0 msgr2=0x555cec7c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7c8fa0 0x555cec7c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.343+0000 7f782259c640 1 -- 172.21.15.16:0/3626223961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cec835370 con 0x555cec7b8e50 2024-04-25T01:31:06.323 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f7822d9d640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7c8fa0 0x555cec7c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f782259c640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7ca450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78140039a0 tx=0x7f781402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a98a6e28d8654c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f7809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f7809ffb640 1 -- 172.21.15.16:0/3626223961 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814035040 con 0x555cec7b8e50 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f7809ffb640 1 -- 172.21.15.16:0/3626223961 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7814003b90 con 0x555cec7b8e50 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f780c09bcd0 con 0x555cec7b8e50 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f7809ffb640 1 -- 172.21.15.16:0/3626223961 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7814003d90 con 0x555cec7b8e50 2024-04-25T01:31:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f782b9b1740 10 client.25225 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.344+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555cec8365a0 con 0x555cec7b8e50 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f7809ffb640 1 -- 172.21.15.16:0/3626223961 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f780c09bcd0 con 0x555cec7b8e50 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f7809ffb640 1 -- 172.21.15.16:0/3626223961 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f78140318f0 con 0x555cec7b8e50 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f782b9b1740 10 client.25225 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f782b9b1740 10 client.25225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f782b9b1740 20 client.25225 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec836ed0 0x555cec8392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f782b9b1740 4 client.25225 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cec72a1f0 con 0x555cec836ed0 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.345+0000 7f781bfff640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec836ed0 0x555cec8392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.346+0000 7f781bfff640 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec836ed0 0x555cec8392f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5725045521ec1f8e server_cookie=4050167a92a8777a in_seq=0 out_seq=0 2024-04-25T01:31:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.346+0000 7f7809ffb640 10 client.25225 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:06.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f7809ffb640 1 -- 172.21.15.16:0/3626223961 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cec72a1f0 con 0x555cec836ed0 2024-04-25T01:31:06.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f7809ffb640 10 client.25225 handle_command_reply: tid=0 2024-04-25T01:31:06.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f782b9b1740 1 client.25225 shutdown 2024-04-25T01:31:06.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 msgr2=0x555cec7ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7ca450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78140039a0 tx=0x7f781402a410 comp rx=0 tx=0).stop 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 shutdown_connections 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555cec7c8fa0 0x555cec7c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.350+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x555cec836ed0 0x555cec8392f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cec7bb370 0x555cec7c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f7809ffb640 0 client.25225 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f782b9b1740 1 --2- 172.21.15.16:0/3626223961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cec7b8e50 0x555cec7ca450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 >> 172.21.15.16:0/3626223961 conn(0x555cec7b44c0 msgr2=0x555cec7b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 shutdown_connections 2024-04-25T01:31:06.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f782b9b1740 1 -- 172.21.15.16:0/3626223961 wait complete. 2024-04-25T01:31:06.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.351+0000 7f782b9b1740 20 client.25225 trim_cache size 0 max 16384 2024-04-25T01:31:06.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:31:06.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:31:06.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:31:06.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78100a2cb0 msgr2=0x7f78100a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78100a2cb0 0x7f78100a5170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7814005ab0 tx=0x7f7814005a20 comp rx=0 tx=0).stop 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 msgr2=0x7f782419e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f782419e4c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f780c00e9f0 tx=0x7f780c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 shutdown_connections 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f78100a2cb0 0x7f78100a5170 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7814005ab0 tx=0x7f7814005a20 comp rx=0 tx=0).stop 2024-04-25T01:31:06.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109980 0x7f78241a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7824108fc0 0x7f782419e4c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 --2- 172.21.15.16:0/1061921153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7824107dd0 0x7f782419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.352+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 >> 172.21.15.16:0/1061921153 conn(0x7f7824076d90 msgr2=0x7f782410c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.353+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 shutdown_connections 2024-04-25T01:31:06.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:06.353+0000 7f7828ca5640 1 -- 172.21.15.16:0/1061921153 wait complete. 2024-04-25T01:31:06.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:06 smithi156 ceph-mon[32631]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-25T01:31:07.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:06 smithi016 ceph-mon[29153]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-25T01:31:07.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:06 smithi038 ceph-mon[32841]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-25T01:31:09.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:08 smithi016 ceph-mon[29153]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:31:09.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:08 smithi038 ceph-mon[32841]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:31:09.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:08 smithi156 ceph-mon[32631]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:31:11.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:10 smithi156 ceph-mon[32631]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:11.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:10 smithi016 ceph-mon[29153]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:11.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:10 smithi038 ceph-mon[32841]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:13.015 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:12 smithi038 ceph-mon[32841]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:13.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:12 smithi016 ceph-mon[29153]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:13.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:12 smithi156 ceph-mon[32631]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:15.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:14 smithi016 ceph-mon[29153]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T01:31:15.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:14 smithi038 ceph-mon[32841]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T01:31:15.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:14 smithi156 ceph-mon[32631]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T01:31:17.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:16 smithi016 ceph-mon[29153]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:17.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:16 smithi038 ceph-mon[32841]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:17.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:16 smithi156 ceph-mon[32631]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:19.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:18 smithi016 ceph-mon[29153]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:31:19.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:18 smithi038 ceph-mon[32841]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:31:19.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:18 smithi156 ceph-mon[32631]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-04-25T01:31:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 -- 172.21.15.16:0/2380377255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 msgr2=0x7fea4c10dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 --2- 172.21.15.16:0/2380377255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c10dee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fea44009ca0 tx=0x7fea4402a8b0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 -- 172.21.15.16:0/2380377255 shutdown_connections 2024-04-25T01:31:19.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 --2- 172.21.15.16:0/2380377255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c10dee0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 --2- 172.21.15.16:0/2380377255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea4c10b0b0 0x7fea4c10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 --2- 172.21.15.16:0/2380377255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea4c108740 0x7fea4c10ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 -- 172.21.15.16:0/2380377255 >> 172.21.15.16:0/2380377255 conn(0x7fea4c06e7b0 msgr2=0x7fea4c070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 -- 172.21.15.16:0/2380377255 shutdown_connections 2024-04-25T01:31:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.500+0000 7fea50929640 1 -- 172.21.15.16:0/2380377255 wait complete. 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.501+0000 7fea50929640 1 Processor -- start 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 -- start start 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea4c108740 0x7fea4c1a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea4c10b0b0 0x7fea4c1a6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c1adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c11c130 con 0x7fea4c108740 2024-04-25T01:31:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c11bfb0 con 0x7fea4c10da60 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea50929640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c11c2b0 con 0x7fea4c10b0b0 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c1adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c1adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55370/0 (socket says 172.21.15.16:55370) 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea4ad9d640 1 -- 172.21.15.16:0/330606282 learned_addr learned my addr 172.21.15.16:0/330606282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea4a59c640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea4c108740 0x7fea4c1a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.502+0000 7fea49d9b640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea4c10b0b0 0x7fea4c1a6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea4ad9d640 1 -- 172.21.15.16:0/330606282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea4c10b0b0 msgr2=0x7fea4c1a6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea4ad9d640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea4c10b0b0 0x7fea4c1a6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea4ad9d640 1 -- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea4c108740 msgr2=0x7fea4c1a6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea4ad9d640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea4c108740 0x7fea4c1a6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea4ad9d640 1 -- 172.21.15.16:0/330606282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c1ae230 con 0x7fea4c10da60 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea4ad9d640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c1adbc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea44003ba0 tx=0x7fea44003bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea3b7fe640 1 -- 172.21.15.16:0/330606282 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44004240 con 0x7fea4c10da60 2024-04-25T01:31:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea3b7fe640 1 -- 172.21.15.16:0/330606282 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea44040070 con 0x7fea4c10da60 2024-04-25T01:31:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.503+0000 7fea3b7fe640 1 -- 172.21.15.16:0/330606282 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44031b90 con 0x7fea4c10da60 2024-04-25T01:31:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.504+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4c1ae4c0 con 0x7fea4c10da60 2024-04-25T01:31:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.504+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4c1ae9c0 con 0x7fea4c10da60 2024-04-25T01:31:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.508+0000 7fea53638740 1 Processor -- start 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.508+0000 7fea3b7fe640 1 -- 172.21.15.16:0/330606282 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fea440043e0 con 0x7fea4c10da60 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 -- start start 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c2950e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 0x561c29513340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c29515e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c29471cb0 con 0x561c295139b0 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561c2941a5b0 con 0x561c2950de50 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea53638740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561c2946dc70 con 0x561c2950e840 2024-04-25T01:31:19.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.509+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 0x561c29513340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 0x561c29513340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38054/0 (socket says 172.21.15.16:38054) 2024-04-25T01:31:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea4a59c640 1 -- 172.21.15.16:0/2363938072 learned_addr learned my addr 172.21.15.16:0/2363938072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea49d9b640 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c29515e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea4a59c640 1 -- 172.21.15.16:0/2363938072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 msgr2=0x561c2950e270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea4a59c640 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c2950e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea4a59c640 1 -- 172.21.15.16:0/2363938072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 msgr2=0x561c29515e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.510+0000 7fea3b7fe640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea340a2d30 0x7fea340a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.511+0000 7fea4a59c640 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c29515e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.511+0000 7fea4a59c640 1 -- 172.21.15.16:0/2363938072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c29186210 con 0x561c2950e840 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.511+0000 7fea3b7fe640 1 -- 172.21.15.16:0/330606282 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea440f0320 con 0x7fea4c10da60 2024-04-25T01:31:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.511+0000 7fea49d9b640 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c29515e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.511+0000 7fea49d9b640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea340a2d30 0x7fea340a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.512+0000 7fea4a59c640 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 0x561c29513340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009cf0 tx=0x7fea3c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba074b9201ca5a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.512+0000 7fea49d9b640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea340a2d30 0x7fea340a51f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x561c2950f0a0 tx=0x7fea400046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.512+0000 7fea38ff9640 1 -- 172.21.15.16:0/2363938072 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c004230 con 0x561c2950e840 2024-04-25T01:31:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.513+0000 7fea38ff9640 1 -- 172.21.15.16:0/2363938072 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea3c0043d0 con 0x561c2950e840 2024-04-25T01:31:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.513+0000 7fea38ff9640 1 -- 172.21.15.16:0/2363938072 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c004230 con 0x561c2950e840 2024-04-25T01:31:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.514+0000 7fea53638740 1 -- 172.21.15.16:0/2363938072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 msgr2=0x561c29513340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.514+0000 7fea53638740 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 0x561c29513340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009cf0 tx=0x7fea3c011810 comp rx=0 tx=0).stop 2024-04-25T01:31:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 -- 172.21.15.16:0/2363938072 shutdown_connections 2024-04-25T01:31:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c29515e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950e840 0x561c29513340 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 --2- 172.21.15.16:0/2363938072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c2950e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 -- 172.21.15.16:0/2363938072 >> 172.21.15.16:0/2363938072 conn(0x561c295094c0 msgr2=0x561c295098c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 -- 172.21.15.16:0/2363938072 shutdown_connections 2024-04-25T01:31:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.515+0000 7fea53638740 1 -- 172.21.15.16:0/2363938072 wait complete. 2024-04-25T01:31:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 -- start start 2024-04-25T01:31:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 Processor -- start 2024-04-25T01:31:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c29529260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c295297a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c295307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c29527970 con 0x561c295139b0 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561c2942db50 con 0x561c2950e840 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c295297a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c295307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c29529260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c29529260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38068/0 (socket says 172.21.15.16:38068) 2024-04-25T01:31:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c295297a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55386/0 (socket says 172.21.15.16:55386) 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea4ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c295307e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43414/0 (socket says 172.21.15.16:43414) 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea4a59c640 1 -- 172.21.15.16:0/2118734068 learned_addr learned my addr 172.21.15.16:0/2118734068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561c29527af0 con 0x561c2950de50 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.516+0000 7fea53638740 10 client.0 _ll_register_callbacks cb 0x561c29468ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea4a59c640 1 -- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 msgr2=0x561c295297a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea4a59c640 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c295297a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea4a59c640 1 -- 172.21.15.16:0/2118734068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 msgr2=0x561c295307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea4a59c640 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c295307e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea4a59c640 1 -- 172.21.15.16:0/2118734068 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c29252620 con 0x561c2950de50 2024-04-25T01:31:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea3c0043d0 con 0x561c2950de50 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea4a59c640 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c29529260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fea3c01a000 tx=0x7fea3c003bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a250580288456be8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea31ffb640 10 client.25234 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.517+0000 7fea31ffb640 1 -- 172.21.15.16:0/2118734068 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c031050 con 0x561c2950de50 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.518+0000 7fea31ffb640 1 -- 172.21.15.16:0/2118734068 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea3c0046c0 con 0x561c2950de50 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.518+0000 7fea31ffb640 1 -- 172.21.15.16:0/2118734068 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fea3c027070 con 0x561c2950de50 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.518+0000 7fea53638740 10 client.25234 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.518+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=22}) -- 0x561c29533a10 con 0x561c2950de50 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea31ffb640 1 -- 172.21.15.16:0/2118734068 <== mon.1 v2:172.21.15.156:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fea3c004900 con 0x561c2950de50 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea53638740 10 client.25234 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea53638740 10 client.25234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea53638740 20 client.25234 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea53638740 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c29534dd0 0x561c295371c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea53638740 4 client.25234 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c2947f1f0 con 0x561c29534dd0 2024-04-25T01:31:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.519+0000 7fea49d9b640 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c29534dd0 0x561c295371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.520+0000 7fea49d9b640 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c29534dd0 0x561c295371c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d8d3f787259316 server_cookie=46968fbc731450fe in_seq=0 out_seq=0 2024-04-25T01:31:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.520+0000 7fea31ffb640 10 client.25234 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea31ffb640 1 -- 172.21.15.16:0/2118734068 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561c2947f1f0 con 0x561c29534dd0 2024-04-25T01:31:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea31ffb640 10 client.25234 handle_command_reply: tid=0 2024-04-25T01:31:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 client.25234 shutdown 2024-04-25T01:31:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 msgr2=0x561c29529260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c29529260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fea3c01a000 tx=0x7fea3c003bf0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 shutdown_connections 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c295307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c29534dd0 0x561c295371c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c295297a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 --2- 172.21.15.16:0/2118734068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c29529260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 >> 172.21.15.16:0/2118734068 conn(0x561c295094c0 msgr2=0x561c295085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea31ffb640 0 client.25234 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 shutdown_connections 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 1 -- 172.21.15.16:0/2118734068 wait complete. 2024-04-25T01:31:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.521+0000 7fea53638740 20 client.25234 trim_cache size 0 max 16384 2024-04-25T01:31:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.546+0000 7fea53638740 1 Processor -- start 2024-04-25T01:31:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.546+0000 7fea53638740 1 -- start start 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.546+0000 7fea53638740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c29508440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.546+0000 7fea53638740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c2950e840 0x561c29530700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea53638740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c295139b0 0x561c29508980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c29508440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c29508440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55398/0 (socket says 172.21.15.16:55398) 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea53638740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1b1ab0 con 0x561c2950e840 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c2950e840 0x561c29530700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea4a59c640 1 -- 172.21.15.16:0/3404884280 learned_addr learned my addr 172.21.15.16:0/3404884280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea4ad9d640 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c295139b0 0x561c29508980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1af380 con 0x561c2950de50 2024-04-25T01:31:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.547+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c116b60 con 0x561c295139b0 2024-04-25T01:31:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea49d9b640 1 -- 172.21.15.16:0/3404884280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c295139b0 msgr2=0x561c29508980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea49d9b640 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c295139b0 0x561c29508980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea49d9b640 1 -- 172.21.15.16:0/3404884280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 msgr2=0x561c29508440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea49d9b640 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c29508440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea49d9b640 1 -- 172.21.15.16:0/3404884280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c29527850 con 0x561c2950e840 2024-04-25T01:31:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea49d9b640 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c2950e840 0x561c29530700 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x561c295379d0 tx=0x7fea40013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5b83399c9baf04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.548+0000 7fea33fff640 1 -- 172.21.15.16:0/3404884280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4001a1d0 con 0x561c2950e840 2024-04-25T01:31:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.549+0000 7fea33fff640 1 -- 172.21.15.16:0/3404884280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea40029620 con 0x561c2950e840 2024-04-25T01:31:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.549+0000 7fea33fff640 1 -- 172.21.15.16:0/3404884280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40026c80 con 0x561c2950e840 2024-04-25T01:31:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c2950e840 msgr2=0x561c29530700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c2950e840 0x561c29530700 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x561c295379d0 tx=0x7fea40013000 comp rx=0 tx=0).stop 2024-04-25T01:31:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 shutdown_connections 2024-04-25T01:31:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c295139b0 0x561c29508980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c2950e840 0x561c29530700 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 --2- 172.21.15.16:0/3404884280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950de50 0x561c29508440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 >> 172.21.15.16:0/3404884280 conn(0x561c295094c0 msgr2=0x561c29507d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 shutdown_connections 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 -- 172.21.15.16:0/3404884280 wait complete. 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.550+0000 7fea53638740 1 -- start start 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea53638740 1 Processor -- start 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea53638740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c2952da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea53638740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c2952dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c2952da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea4a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c2952da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38078/0 (socket says 172.21.15.16:38078) 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c2952dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea49d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c2952dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55410/0 (socket says 172.21.15.16:55410) 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea4a59c640 1 -- 172.21.15.16:0/294133264 learned_addr learned my addr 172.21.15.16:0/294133264 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c2952efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.551+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1171d0 con 0x561c295139b0 2024-04-25T01:31:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1aece0 con 0x561c2950e840 2024-04-25T01:31:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4ad9d640 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c2952efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1178a0 con 0x561c2950de50 2024-04-25T01:31:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea53638740 10 client.0 _ll_register_callbacks cb 0x7fea3c009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4a59c640 1 -- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 msgr2=0x561c2952dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4a59c640 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c2952dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4a59c640 1 -- 172.21.15.16:0/294133264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 msgr2=0x561c2952efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4a59c640 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c2952efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4a59c640 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c2952f5a0 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.552+0000 7fea4a59c640 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c2952da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009530 tx=0x7fea3c022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614688db366d4c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea31ffb640 1 -- 172.21.15.16:0/294133264 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c031040 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea31ffb640 1 -- 172.21.15.16:0/294133264 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea3c005630 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea40029620 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea31ffb640 1 -- 172.21.15.16:0/294133264 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fea3c004230 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea53638740 10 client.25240 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561c29530890 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.553+0000 7fea31ffb640 1 -- 172.21.15.16:0/294133264 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea3c0043d0 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea31ffb640 1 -- 172.21.15.16:0/294133264 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fea3c041930 con 0x561c2950de50 2024-04-25T01:31:19.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea53638740 10 client.25240 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea53638740 10 client.25240 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea53638740 20 client.25240 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c2959b8c0 0x561c2959dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea53638740 4 client.25240 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fea40029620 con 0x561c2959b8c0 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.554+0000 7fea49d9b640 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c2959b8c0 0x561c2959dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.555+0000 7fea49d9b640 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c2959b8c0 0x561c2959dcb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33243f4754dc98ed server_cookie=e4b78cd60a23f49a in_seq=0 out_seq=0 2024-04-25T01:31:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.555+0000 7fea31ffb640 10 client.25240 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.558+0000 7fea31ffb640 1 -- 172.21.15.16:0/294133264 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fea40029620 con 0x561c2959b8c0 2024-04-25T01:31:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.558+0000 7fea31ffb640 10 client.25240 handle_command_reply: tid=0 2024-04-25T01:31:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.558+0000 7fea53638740 1 client.25240 shutdown 2024-04-25T01:31:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 msgr2=0x561c2952da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c2952da60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009530 tx=0x7fea3c022000 comp rx=0 tx=0).stop 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 shutdown_connections 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c295139b0 0x561c2952efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x561c2959b8c0 0x561c2959dcb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561c2950e840 0x561c2952dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 --2- 172.21.15.16:0/294133264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c2950de50 0x561c2952da60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 >> 172.21.15.16:0/294133264 conn(0x561c295094c0 msgr2=0x561c295085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea31ffb640 0 client.25240 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 shutdown_connections 2024-04-25T01:31:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.559+0000 7fea53638740 1 -- 172.21.15.16:0/294133264 wait complete. 2024-04-25T01:31:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.560+0000 7fea53638740 20 client.25240 trim_cache size 0 max 16384 2024-04-25T01:31:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:31:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:31:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:31:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:31:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea340a2d30 msgr2=0x7fea340a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea340a2d30 0x7fea340a51f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x561c2950f0a0 tx=0x7fea400046c0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 msgr2=0x7fea4c1adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c1adbc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea44003ba0 tx=0x7fea44003bd0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 shutdown_connections 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea340a2d30 0x7fea340a51f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea4c10da60 0x7fea4c1adbc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea4c10b0b0 0x7fea4c1a6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 --2- 172.21.15.16:0/330606282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea4c108740 0x7fea4c1a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.561+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 >> 172.21.15.16:0/330606282 conn(0x7fea4c06e7b0 msgr2=0x7fea4c10bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.562+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 shutdown_connections 2024-04-25T01:31:19.540 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:19.562+0000 7fea50929640 1 -- 172.21.15.16:0/330606282 wait complete. 2024-04-25T01:31:21.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:20 smithi016 ceph-mon[29153]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:21.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:20 smithi038 ceph-mon[32841]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:21.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:20 smithi156 ceph-mon[32631]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:23.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:22 smithi016 ceph-mon[29153]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:23.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:22 smithi038 ceph-mon[32841]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:23.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:22 smithi156 ceph-mon[32631]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:31:25.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:24 smithi016 ceph-mon[29153]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:31:25.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:24 smithi038 ceph-mon[32841]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:31:25.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:24 smithi156 ceph-mon[32631]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T01:31:27.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:26 smithi016 ceph-mon[29153]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:27.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:26 smithi038 ceph-mon[32841]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:27.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:26 smithi156 ceph-mon[32631]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:29.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:28 smithi016 ceph-mon[29153]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:29.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:28 smithi038 ceph-mon[32841]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:29.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:28 smithi156 ceph-mon[32631]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:31:31.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:30 smithi016 ceph-mon[29153]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T01:31:31.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:30 smithi038 ceph-mon[32841]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T01:31:31.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:30 smithi156 ceph-mon[32631]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T01:31:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.708+0000 7fe7af05b640 1 -- 172.21.15.16:0/870807598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009c370 msgr2=0x7fe7a009c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.708+0000 7fe7af05b640 1 --2- 172.21.15.16:0/870807598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009c370 0x7fe7a009c790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe798009900 tx=0x7fe79802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 -- 172.21.15.16:0/870807598 shutdown_connections 2024-04-25T01:31:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 --2- 172.21.15.16:0/870807598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7a009df20 0x7fe7a00a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 --2- 172.21.15.16:0/870807598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009d560 0x7fe7a009d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 --2- 172.21.15.16:0/870807598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009c370 0x7fe7a009c790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 -- 172.21.15.16:0/870807598 >> 172.21.15.16:0/870807598 conn(0x7fe7a000b310 msgr2=0x7fe7a000b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 -- 172.21.15.16:0/870807598 shutdown_connections 2024-04-25T01:31:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.709+0000 7fe7af05b640 1 -- 172.21.15.16:0/870807598 wait complete. 2024-04-25T01:31:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.711+0000 7fe7af05b640 1 Processor -- start 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 -- start start 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009c370 0x7fe7a0132460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7a009d560 0x7fe7a01329a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009df20 0x7fe7a0139a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a00a7ed0 con 0x7fe7a009d560 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a00a7d50 con 0x7fe7a009df20 2024-04-25T01:31:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.713+0000 7fe7af05b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a00a8050 con 0x7fe7a009c370 2024-04-25T01:31:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae059640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009c370 0x7fe7a0132460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae85a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009df20 0x7fe7a0139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae85a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009df20 0x7fe7a0139a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:56168/0 (socket says 172.21.15.16:56168) 2024-04-25T01:31:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2363562917 learned_addr learned my addr 172.21.15.16:0/2363562917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae059640 1 -- 172.21.15.16:0/2363562917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009df20 msgr2=0x7fe7a0139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009df20 0x7fe7a0139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae059640 1 -- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7a009d560 msgr2=0x7fe7a01329a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7a009d560 0x7fe7a01329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.714+0000 7fe7ae059640 1 -- 172.21.15.16:0/2363562917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a014b090 con 0x7fe7a009c370 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.715+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009c370 0x7fe7a0132460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe79802a910 tx=0x7fe798004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.715+0000 7fe79f7fe640 1 -- 172.21.15.16:0/2363562917 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79803f030 con 0x7fe7a009c370 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.715+0000 7fe79f7fe640 1 -- 172.21.15.16:0/2363562917 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe798003cb0 con 0x7fe7a009c370 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.715+0000 7fe79f7fe640 1 -- 172.21.15.16:0/2363562917 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798035070 con 0x7fe7a009c370 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.716+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a014b320 con 0x7fe7a009c370 2024-04-25T01:31:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.716+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a014b780 con 0x7fe7a009c370 2024-04-25T01:31:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.719+0000 7fe79f7fe640 1 -- 172.21.15.16:0/2363562917 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe798042030 con 0x7fe7a009c370 2024-04-25T01:31:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 Processor -- start 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 -- start start 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f8e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8ea20 0x558732f93520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f93b90 0x558732f96020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558732ef1cb0 con 0x558732f93b90 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558732e9a5b0 con 0x558732f8ea20 2024-04-25T01:31:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.720+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558732eedc70 con 0x558732f8e030 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f8e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f8e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:43040/0 (socket says 172.21.15.16:43040) 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2607950247 learned_addr learned my addr 172.21.15.16:0/2607950247 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe79f7fe640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe7940a2d60 0x7fe7940a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8ea20 0x558732f93520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ad858640 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f93b90 0x558732f96020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe7940a2d60 0x7fe7940a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2607950247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8ea20 msgr2=0x558732f93520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8ea20 0x558732f93520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2607950247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f93b90 msgr2=0x558732f96020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f93b90 0x558732f96020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.721+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2607950247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558732c06210 con 0x558732f8e030 2024-04-25T01:31:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.722+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f8e450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4013ae0 tx=0x7fe7a400f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3586ca7cf2953956 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.722+0000 7fe79f7fe640 1 -- 172.21.15.16:0/2363562917 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe7980ef770 con 0x7fe7a009c370 2024-04-25T01:31:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.722+0000 7fe79cbfe640 1 -- 172.21.15.16:0/2607950247 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4013620 con 0x558732f8e030 2024-04-25T01:31:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.722+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe7940a2d60 0x7fe7940a5220 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400f860 tx=0x7fe7a401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.722+0000 7fe79cbfe640 1 -- 172.21.15.16:0/2607950247 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7a4009d90 con 0x558732f8e030 2024-04-25T01:31:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.722+0000 7fe79cbfe640 1 -- 172.21.15.16:0/2607950247 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4011690 con 0x558732f8e030 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2607950247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 msgr2=0x558732f8e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f8e450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4013ae0 tx=0x7fe7a400f360 comp rx=0 tx=0).stop 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2607950247 shutdown_connections 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f93b90 0x558732f96020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8ea20 0x558732f93520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2607950247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f8e450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.723+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2607950247 >> 172.21.15.16:0/2607950247 conn(0x558732f896a0 msgr2=0x558732f89aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.724+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2607950247 shutdown_connections 2024-04-25T01:31:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.724+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2607950247 wait complete. 2024-04-25T01:31:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.724+0000 7fe7b2fd4740 1 -- start start 2024-04-25T01:31:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 Processor -- start 2024-04-25T01:31:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f93b90 0x558732f9f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558732f96a80 con 0x558732f8ea20 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558732eadb50 con 0x558732f8e030 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7ae059640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7ae059640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:56192/0 (socket says 172.21.15.16:56192) 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7ae059640 1 -- 172.21.15.16:0/2976918122 learned_addr learned my addr 172.21.15.16:0/2976918122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558732f96c00 con 0x558732f93b90 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f93b90 0x558732f9f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.725+0000 7fe7b2fd4740 10 client.0 _ll_register_callbacks cb 0x558732ee8ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ad858640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8e030 msgr2=0x558732f983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2976918122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 msgr2=0x558732f98930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2976918122 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558732cd2620 con 0x558732f93b90 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f93b90 0x558732f9f970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4013ae0 tx=0x7fe7a40116a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac69d8850067637d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.726+0000 7fe791ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe791ffb640 1 -- 172.21.15.16:0/2976918122 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a402c020 con 0x558732f93b90 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7a4009d90 con 0x558732f93b90 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe791ffb640 1 -- 172.21.15.16:0/2976918122 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe7a4013620 con 0x558732f93b90 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe791ffb640 1 -- 172.21.15.16:0/2976918122 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe7a4037050 con 0x558732f93b90 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe7b2fd4740 10 client.25249 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558732fa2d80 con 0x558732f93b90 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe791ffb640 1 -- 172.21.15.16:0/2976918122 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7a4009d90 con 0x558732f93b90 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.727+0000 7fe791ffb640 1 -- 172.21.15.16:0/2976918122 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe7a4005040 con 0x558732f93b90 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7b2fd4740 10 client.25249 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7b2fd4740 10 client.25249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7b2fd4740 20 client.25249 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558732fa48d0 0x558732fa6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7b2fd4740 4 client.25249 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558732eff1f0 con 0x558732fa48d0 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.728+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558732fa48d0 0x558732fa6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.729+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558732fa48d0 0x558732fa6d90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df67c0942b8d1e13 server_cookie=98c448f29a1395d7 in_seq=0 out_seq=0 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.729+0000 7fe791ffb640 10 client.25249 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe791ffb640 1 -- 172.21.15.16:0/2976918122 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558732eff1f0 con 0x558732fa48d0 2024-04-25T01:31:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe791ffb640 10 client.25249 handle_command_reply: tid=0 2024-04-25T01:31:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 client.25249 shutdown 2024-04-25T01:31:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f93b90 msgr2=0x558732f9f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f93b90 0x558732f9f970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4013ae0 tx=0x7fe7a40116a0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 shutdown_connections 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f93b90 0x558732f9f970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558732fa48d0 0x558732fa6d90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.730+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2976918122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.731+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 >> 172.21.15.16:0/2976918122 conn(0x558732f896a0 msgr2=0x558732f888a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.731+0000 7fe791ffb640 0 client.25249 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.731+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 shutdown_connections 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.731+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2976918122 wait complete. 2024-04-25T01:31:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.731+0000 7fe7b2fd4740 20 client.25249 trim_cache size 0 max 16384 2024-04-25T01:31:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.759+0000 7fe7b2fd4740 1 Processor -- start 2024-04-25T01:31:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.759+0000 7fe7b2fd4740 1 -- start start 2024-04-25T01:31:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0156ff0 con 0x558732f8e030 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a01548c0 con 0x558732f93b90 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a01526f0 con 0x558732f8ea20 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:56194/0 (socket says 172.21.15.16:56194) 2024-04-25T01:31:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2146889506 learned_addr learned my addr 172.21.15.16:0/2146889506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ad858640 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2146889506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8ea20 msgr2=0x558732f98930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2146889506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8e030 msgr2=0x558732f983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.760+0000 7fe7ae85a640 1 -- 172.21.15.16:0/2146889506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558732f891e0 con 0x558732f93b90 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.761+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4005970 tx=0x7fe7a40059a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73a49364a53dcfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.761+0000 7fe793fff640 1 -- 172.21.15.16:0/2146889506 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a403f020 con 0x558732f93b90 2024-04-25T01:31:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.761+0000 7fe793fff640 1 -- 172.21.15.16:0/2146889506 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7a4009d70 con 0x558732f93b90 2024-04-25T01:31:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.761+0000 7fe793fff640 1 -- 172.21.15.16:0/2146889506 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a402c070 con 0x558732f93b90 2024-04-25T01:31:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.762+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2146889506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 msgr2=0x558732f9f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.762+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4005970 tx=0x7fe7a40059a0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2146889506 shutdown_connections 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f930 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8ea20 0x558732f98930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2146889506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8e030 0x558732f983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2146889506 >> 172.21.15.16:0/2146889506 conn(0x558732f896a0 msgr2=0x558732fa68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2146889506 shutdown_connections 2024-04-25T01:31:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.763+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2146889506 wait complete. 2024-04-25T01:31:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.764+0000 7fe7b2fd4740 1 -- start start 2024-04-25T01:31:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.764+0000 7fe7b2fd4740 1 Processor -- start 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f9bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f9c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7ae059640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f9bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7ae059640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f9bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:43060/0 (socket says 172.21.15.16:43060) 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0152d80 con 0x558732f8ea20 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a01541f0 con 0x558732f93b90 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0153450 con 0x558732f8e030 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7b2fd4740 10 client.0 _ll_register_callbacks cb 0x7fe7a8067bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7ae059640 1 -- 172.21.15.16:0/2468077478 learned_addr learned my addr 172.21.15.16:0/2468077478 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.765+0000 7fe7ae85a640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ad858640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f9c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ae059640 1 -- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 msgr2=0x558732f9f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ae059640 1 -- 172.21.15.16:0/2468077478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 msgr2=0x558732f9c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f9c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ae059640 1 -- 172.21.15.16:0/2468077478 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558732f9f690 con 0x558732f8e030 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.766+0000 7fe7ae059640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f9bbb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe7980bbee0 tx=0x7fe79802a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0af743e7b36aa0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe791ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe791ffb640 1 -- 172.21.15.16:0/2468077478 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798049040 con 0x558732f8e030 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe791ffb640 1 -- 172.21.15.16:0/2468077478 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe79803bd60 con 0x558732f8e030 2024-04-25T01:31:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7a4009d70 con 0x558732f8e030 2024-04-25T01:31:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe791ffb640 1 -- 172.21.15.16:0/2468077478 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe7980bc650 con 0x558732f8e030 2024-04-25T01:31:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe7b2fd4740 10 client.25255 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.767+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558733009aa0 con 0x558732f8e030 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe791ffb640 1 -- 172.21.15.16:0/2468077478 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7a4009d70 con 0x558732f8e030 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe791ffb640 1 -- 172.21.15.16:0/2468077478 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe79803b4f0 con 0x558732f8e030 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7b2fd4740 10 client.25255 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7b2fd4740 10 client.25255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7b2fd4740 20 client.25255 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55873300a440 0x55873300c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7b2fd4740 4 client.25255 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558732ef9540 con 0x55873300a440 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.768+0000 7fe7ad858640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55873300a440 0x55873300c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.769+0000 7fe7ad858640 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55873300a440 0x55873300c860 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94dc08741ff031c4 server_cookie=844e83d2ec29259a in_seq=0 out_seq=0 2024-04-25T01:31:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.769+0000 7fe791ffb640 10 client.25255 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.773+0000 7fe791ffb640 1 -- 172.21.15.16:0/2468077478 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558732ef9540 con 0x55873300a440 2024-04-25T01:31:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.773+0000 7fe791ffb640 10 client.25255 handle_command_reply: tid=0 2024-04-25T01:31:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.773+0000 7fe7b2fd4740 1 client.25255 shutdown 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 msgr2=0x558732f9bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f9bbb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe7980bbee0 tx=0x7fe79802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 shutdown_connections 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55873300a440 0x55873300c860 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558732f93b90 0x558732f9f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558732f8ea20 0x558732f9c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 --2- 172.21.15.16:0/2468077478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558732f8e030 0x558732f9bbb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 >> 172.21.15.16:0/2468077478 conn(0x558732f896a0 msgr2=0x558732f8fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe791ffb640 0 client.25255 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.774+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 shutdown_connections 2024-04-25T01:31:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.775+0000 7fe7b2fd4740 1 -- 172.21.15.16:0/2468077478 wait complete. 2024-04-25T01:31:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.775+0000 7fe7b2fd4740 20 client.25255 trim_cache size 0 max 16384 2024-04-25T01:31:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:31:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:31:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:31:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:31:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.775+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe7940a2d60 msgr2=0x7fe7940a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.775+0000 7fe7af05b640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe7940a2d60 0x7fe7940a5220 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400f860 tx=0x7fe7a401f000 comp rx=0 tx=0).stop 2024-04-25T01:31:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.775+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009c370 msgr2=0x7fe7a0132460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.775+0000 7fe7af05b640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009c370 0x7fe7a0132460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe79802a910 tx=0x7fe798004360 comp rx=0 tx=0).stop 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 shutdown_connections 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe7940a2d60 0x7fe7940a5220 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7a009df20 0x7fe7a0139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7a009d560 0x7fe7a01329a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 --2- 172.21.15.16:0/2363562917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7a009c370 0x7fe7a0132460 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 >> 172.21.15.16:0/2363562917 conn(0x7fe7a000b310 msgr2=0x7fe7a00a0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 shutdown_connections 2024-04-25T01:31:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:32.776+0000 7fe7af05b640 1 -- 172.21.15.16:0/2363562917 wait complete. 2024-04-25T01:31:33.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:32 smithi016 ceph-mon[29153]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:31:33.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:32 smithi038 ceph-mon[32841]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:31:33.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:32 smithi156 ceph-mon[32631]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:31:35.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:34 smithi016 ceph-mon[29153]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-25T01:31:35.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:34 smithi038 ceph-mon[32841]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-25T01:31:35.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:34 smithi156 ceph-mon[32631]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-25T01:31:37.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:36 smithi016 ceph-mon[29153]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:37.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:36 smithi038 ceph-mon[32841]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:37.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:36 smithi156 ceph-mon[32631]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T01:31:39.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:38 smithi016 ceph-mon[29153]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-04-25T01:31:39.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:38 smithi038 ceph-mon[32841]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-04-25T01:31:39.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:38 smithi156 ceph-mon[32631]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-04-25T01:31:41.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:40 smithi156 ceph-mon[32631]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T01:31:41.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:40 smithi016 ceph-mon[29153]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T01:31:41.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:40 smithi038 ceph-mon[32841]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T01:31:43.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:42 smithi016 ceph-mon[29153]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:31:43.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:42 smithi038 ceph-mon[32841]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:31:43.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:42 smithi156 ceph-mon[32631]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:31:45.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:44 smithi016 ceph-mon[29153]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-25T01:31:45.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:44 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:45.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:44 smithi038 ceph-mon[32841]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-25T01:31:45.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:44 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:45.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:44 smithi156 ceph-mon[32631]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-25T01:31:45.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:44 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.923+0000 7f90de74f640 1 -- 172.21.15.16:0/2002440207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d8102270 msgr2=0x7f90d8108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.923+0000 7f90de74f640 1 --2- 172.21.15.16:0/2002440207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d8102270 0x7f90d8108b00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009ca0 tx=0x7f90d002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 -- 172.21.15.16:0/2002440207 shutdown_connections 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 --2- 172.21.15.16:0/2002440207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d8102270 0x7f90d8108b00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 --2- 172.21.15.16:0/2002440207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d81018b0 0x7f90d8101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 --2- 172.21.15.16:0/2002440207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81006c0 0x7f90d8100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 -- 172.21.15.16:0/2002440207 >> 172.21.15.16:0/2002440207 conn(0x7f90d80fbc50 msgr2=0x7f90d80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 -- 172.21.15.16:0/2002440207 shutdown_connections 2024-04-25T01:31:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 -- 172.21.15.16:0/2002440207 wait complete. 2024-04-25T01:31:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 Processor -- start 2024-04-25T01:31:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.924+0000 7f90de74f640 1 -- start start 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90de74f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d81006c0 0x7f90d8073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90de74f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 0x7f90d806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90de74f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d8102270 0x7f90d8070060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90de74f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90d810a640 con 0x7f90d81006c0 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90d77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 0x7f90d806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d81006c0 0x7f90d8073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90d77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 0x7f90d806fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43158/0 (socket says 172.21.15.16:43158) 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90d77fe640 1 -- 172.21.15.16:0/3167249527 learned_addr learned my addr 172.21.15.16:0/3167249527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90d810a4c0 con 0x7f90d81018b0 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.925+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f90d810a7c0 con 0x7f90d8102270 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90dccec640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d8102270 0x7f90d8070060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90d77fe640 1 -- 172.21.15.16:0/3167249527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d8102270 msgr2=0x7f90d8070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d8102270 0x7f90d8070060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90d77fe640 1 -- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d81006c0 msgr2=0x7f90d8073060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d81006c0 0x7f90d8073060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90d77fe640 1 -- 172.21.15.16:0/3167249527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d8070670 con 0x7f90d81018b0 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.926+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d81006c0 0x7f90d8073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.927+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 0x7f90d806fb20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00e9f0 tx=0x7f90cc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.927+0000 7f90d57fa640 1 -- 172.21.15.16:0/3167249527 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00cd80 con 0x7f90d81018b0 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.927+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d81100d0 con 0x7f90d81018b0 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.927+0000 7f90d57fa640 1 -- 172.21.15.16:0/3167249527 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90cc0054d0 con 0x7f90d81018b0 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.927+0000 7f90d57fa640 1 -- 172.21.15.16:0/3167249527 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc010430 con 0x7f90d81018b0 2024-04-25T01:31:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.927+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d81aeb70 con 0x7f90d81018b0 2024-04-25T01:31:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.930+0000 7f90d57fa640 1 -- 172.21.15.16:0/3167249527 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f90cc005020 con 0x7f90d81018b0 2024-04-25T01:31:45.909 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90d57fa640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2cb0 0x7f90c00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 Processor -- start 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90d57fa640 1 -- 172.21.15.16:0/3167249527 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f90cc0cf410 con 0x7f90d81018b0 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2cb0 0x7f90c00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 -- start start 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc708e50 0x55f7cc709270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc70e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc710e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc66ccb0 con 0x55f7cc70e9b0 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc6155b0 con 0x55f7cc709840 2024-04-25T01:31:45.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.931+0000 7f90e145c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc668c70 con 0x55f7cc708e50 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc70e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc70e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43170/0 (socket says 172.21.15.16:43170) 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 -- 172.21.15.16:0/3745968586 learned_addr learned my addr 172.21.15.16:0/3745968586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90dccec640 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc710e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2cb0 0x7f90c00a5170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f90d8101710 tx=0x7f90c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc708e50 0x55f7cc709270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 -- 172.21.15.16:0/3745968586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc708e50 msgr2=0x55f7cc709270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc708e50 0x55f7cc709270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 -- 172.21.15.16:0/3745968586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 msgr2=0x55f7cc710e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc710e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 -- 172.21.15.16:0/3745968586 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7cc381210 con 0x55f7cc709840 2024-04-25T01:31:45.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90dccec640 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc710e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:45.912 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.932+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc70e340 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0980b0 tx=0x7f90cc09b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372cb195d1cbd341 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.933+0000 7f90be7fc640 1 -- 172.21.15.16:0/3745968586 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc0b1070 con 0x55f7cc709840 2024-04-25T01:31:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.933+0000 7f90be7fc640 1 -- 172.21.15.16:0/3745968586 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90cc0b4e50 con 0x55f7cc709840 2024-04-25T01:31:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.933+0000 7f90be7fc640 1 -- 172.21.15.16:0/3745968586 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc0a0000 con 0x55f7cc709840 2024-04-25T01:31:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 -- 172.21.15.16:0/3745968586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 msgr2=0x55f7cc70e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc70e340 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0980b0 tx=0x7f90cc09b5b0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 -- 172.21.15.16:0/3745968586 shutdown_connections 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc710e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc70e340 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 --2- 172.21.15.16:0/3745968586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc708e50 0x55f7cc709270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 -- 172.21.15.16:0/3745968586 >> 172.21.15.16:0/3745968586 conn(0x55f7cc7044c0 msgr2=0x55f7cc7048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 -- 172.21.15.16:0/3745968586 shutdown_connections 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 -- 172.21.15.16:0/3745968586 wait complete. 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.934+0000 7f90e145c740 1 -- start start 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.935+0000 7f90e145c740 1 Processor -- start 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.935+0000 7f90e145c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc7132a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.935+0000 7f90e145c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 0x55f7cc7137e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.935+0000 7f90e145c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc71a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90e145c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc711960 con 0x55f7cc70e9b0 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90e145c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc628b50 con 0x55f7cc708e50 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90e145c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc711ae0 con 0x55f7cc709840 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90e145c740 10 client.0 _ll_register_callbacks cb 0x55f7cc663ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc7132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 0x55f7cc7137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90dccec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc71a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 0x55f7cc7137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38372/0 (socket says 172.21.15.16:38372) 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc7132a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43174/0 (socket says 172.21.15.16:43174) 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 -- 172.21.15.16:0/3915659625 learned_addr learned my addr 172.21.15.16:0/3915659625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 -- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 msgr2=0x55f7cc7132a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc7132a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 -- 172.21.15.16:0/3915659625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 msgr2=0x55f7cc71a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc71a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d77fe640 1 -- 172.21.15.16:0/3915659625 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7cc44d620 con 0x55f7cc709840 2024-04-25T01:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc7132a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.936+0000 7f90dccec640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc71a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.937+0000 7f90d77fe640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 0x55f7cc7137e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0cf730 tx=0x7f90cc097f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c1e44637dc10a88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:45.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.937+0000 7f90b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.937+0000 7f90b77fe640 1 -- 172.21.15.16:0/3915659625 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc07e020 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.937+0000 7f90b77fe640 1 -- 172.21.15.16:0/3915659625 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f90cc0a4430 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.937+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90cc0b4e50 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.938+0000 7f90b77fe640 1 -- 172.21.15.16:0/3915659625 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f90cc0b1070 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.938+0000 7f90e145c740 10 client.25267 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.938+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f7cc7119b0 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90b77fe640 1 -- 172.21.15.16:0/3915659625 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90cc09c350 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90b77fe640 1 -- 172.21.15.16:0/3915659625 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f90cc0979a0 con 0x55f7cc709840 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90e145c740 10 client.25267 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90e145c740 10 client.25267 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90e145c740 20 client.25267 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90e145c740 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc71ee90 0x55f7cc721280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90e145c740 4 client.25267 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.939+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f90cc0b4e50 con 0x55f7cc71ee90 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.940+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc71ee90 0x55f7cc721280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.940+0000 7f90d7fff640 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc71ee90 0x55f7cc721280 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d641033f3377db1e server_cookie=6a90067000eee1ed in_seq=0 out_seq=0 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.940+0000 7f90b77fe640 10 client.25267 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.941+0000 7f90b77fe640 1 -- 172.21.15.16:0/3915659625 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f90cc0b4e50 con 0x55f7cc71ee90 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.941+0000 7f90b77fe640 10 client.25267 handle_command_reply: tid=0 2024-04-25T01:31:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.941+0000 7f90e145c740 1 client.25267 shutdown 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 msgr2=0x55f7cc7137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 0x55f7cc7137e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0cf730 tx=0x7f90cc097f80 comp rx=0 tx=0).stop 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 shutdown_connections 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc70e9b0 0x55f7cc71a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc709840 0x55f7cc7137e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc71ee90 0x55f7cc721280 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 --2- 172.21.15.16:0/3915659625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc7132a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 >> 172.21.15.16:0/3915659625 conn(0x55f7cc7044c0 msgr2=0x55f7cc7035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90b77fe640 0 client.25267 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:45.923 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 shutdown_connections 2024-04-25T01:31:45.923 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.942+0000 7f90e145c740 1 -- 172.21.15.16:0/3915659625 wait complete. 2024-04-25T01:31:45.923 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.943+0000 7f90e145c740 20 client.25267 trim_cache size 0 max 16384 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.966+0000 7f90e145c740 1 Processor -- start 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.966+0000 7f90e145c740 1 -- start start 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90e145c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc707530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90e145c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc709840 0x55f7cc71a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90e145c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc707a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90e145c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90d81ba3a0 con 0x55f7cc709840 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90e145c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90d81b7c70 con 0x55f7cc708e50 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90e145c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f90d81b5ac0 con 0x55f7cc70e9b0 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc707530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.945 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc707a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc707a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38376/0 (socket says 172.21.15.16:38376) 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 -- 172.21.15.16:0/2764653305 learned_addr learned my addr 172.21.15.16:0/2764653305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90d77fe640 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc709840 0x55f7cc71a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 -- 172.21.15.16:0/2764653305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 msgr2=0x55f7cc707530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc707530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 -- 172.21.15.16:0/2764653305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc709840 msgr2=0x55f7cc71a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc709840 0x55f7cc71a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.967+0000 7f90dccec640 1 -- 172.21.15.16:0/2764653305 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7cc709c20 con 0x55f7cc70e9b0 2024-04-25T01:31:45.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.968+0000 7f90dccec640 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc707a70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009ca0 tx=0x7f90d00098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0c168e7709939db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:45.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.968+0000 7f90bdffb640 1 -- 172.21.15.16:0/2764653305 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0031d40 con 0x55f7cc70e9b0 2024-04-25T01:31:45.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.968+0000 7f90bdffb640 1 -- 172.21.15.16:0/2764653305 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90d004b070 con 0x55f7cc70e9b0 2024-04-25T01:31:45.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.968+0000 7f90bdffb640 1 -- 172.21.15.16:0/2764653305 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d00305c0 con 0x55f7cc70e9b0 2024-04-25T01:31:45.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.969+0000 7f90e145c740 1 -- 172.21.15.16:0/2764653305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 msgr2=0x55f7cc707a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.969+0000 7f90e145c740 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc707a70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009ca0 tx=0x7f90d00098b0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.969+0000 7f90e145c740 1 -- 172.21.15.16:0/2764653305 shutdown_connections 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.969+0000 7f90e145c740 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc707a70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.969+0000 7f90e145c740 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc709840 0x55f7cc71a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.969+0000 7f90e145c740 1 --2- 172.21.15.16:0/2764653305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc708e50 0x55f7cc707530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.970+0000 7f90e145c740 1 -- 172.21.15.16:0/2764653305 >> 172.21.15.16:0/2764653305 conn(0x55f7cc7044c0 msgr2=0x55f7cc70aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.970+0000 7f90e145c740 1 -- 172.21.15.16:0/2764653305 shutdown_connections 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.970+0000 7f90e145c740 1 -- 172.21.15.16:0/2764653305 wait complete. 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.970+0000 7f90e145c740 1 -- start start 2024-04-25T01:31:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 Processor -- start 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 0x55f7cc717ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc718020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 0x55f7cc717ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc719030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 0x55f7cc717ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36416/0 (socket says 172.21.15.16:36416) 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90d7fff640 1 -- 172.21.15.16:0/1655447526 learned_addr learned my addr 172.21.15.16:0/1655447526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90d81b6130 con 0x55f7cc708e50 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90d81b75a0 con 0x55f7cc709840 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.971+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f90d81b6800 con 0x55f7cc70e9b0 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90e145c740 10 client.0 _ll_register_callbacks cb 0x7f90cc09baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d77fe640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc718020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90dccec640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc719030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d7fff640 1 -- 172.21.15.16:0/1655447526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 msgr2=0x55f7cc719030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d7fff640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc719030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d7fff640 1 -- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 msgr2=0x55f7cc718020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d7fff640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc718020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d7fff640 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7cc719620 con 0x55f7cc708e50 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90d7fff640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 0x55f7cc717ae0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90c8015000 tx=0x7f90c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17947c97cce33c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.972+0000 7f90b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90b77fe640 1 -- 172.21.15.16:0/1655447526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c802a6f0 con 0x55f7cc708e50 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90b77fe640 1 -- 172.21.15.16:0/1655447526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f90c802a890 con 0x55f7cc708e50 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d004b070 con 0x55f7cc708e50 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90b77fe640 1 -- 172.21.15.16:0/1655447526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f90c8039050 con 0x55f7cc708e50 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90e145c740 10 client.15618 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f7cc71a9e0 con 0x55f7cc708e50 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90b77fe640 1 -- 172.21.15.16:0/1655447526 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90c802e020 con 0x55f7cc708e50 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.973+0000 7f90b77fe640 1 -- 172.21.15.16:0/1655447526 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f90c800f040 con 0x55f7cc708e50 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90e145c740 10 client.15618 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90e145c740 10 client.15618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90e145c740 20 client.15618 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90e145c740 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc785a60 0x55f7cc787e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90e145c740 4 client.15618 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f90d004b070 con 0x55f7cc785a60 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90d77fe640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc785a60 0x55f7cc787e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90b77fe640 10 client.15618 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.974+0000 7f90d77fe640 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc785a60 0x55f7cc787e80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fb18c3cc4168544 server_cookie=34c8d3b30662842f in_seq=0 out_seq=0 2024-04-25T01:31:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.983+0000 7f90b77fe640 1 -- 172.21.15.16:0/1655447526 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f90d004b070 con 0x55f7cc785a60 2024-04-25T01:31:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.983+0000 7f90b77fe640 10 client.15618 handle_command_reply: tid=0 2024-04-25T01:31:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.983+0000 7f90e145c740 1 client.15618 shutdown 2024-04-25T01:31:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 msgr2=0x55f7cc717ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 0x55f7cc717ae0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90c8015000 tx=0x7f90c8004730 comp rx=0 tx=0).stop 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 shutdown_connections 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7cc70e9b0 0x55f7cc719030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f7cc785a60 0x55f7cc787e80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7cc709840 0x55f7cc718020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 --2- 172.21.15.16:0/1655447526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7cc708e50 0x55f7cc717ae0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90b77fe640 0 client.15618 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 >> 172.21.15.16:0/1655447526 conn(0x55f7cc7044c0 msgr2=0x55f7cc70aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 shutdown_connections 2024-04-25T01:31:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.984+0000 7f90e145c740 1 -- 172.21.15.16:0/1655447526 wait complete. 2024-04-25T01:31:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.985+0000 7f90e145c740 20 client.15618 trim_cache size 0 max 16384 2024-04-25T01:31:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:31:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:31:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:31:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.985+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2cb0 msgr2=0x7f90c00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.985+0000 7f90de74f640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2cb0 0x7f90c00a5170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f90d8101710 tx=0x7f90c8005f90 comp rx=0 tx=0).stop 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.986+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 msgr2=0x7f90d806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.986+0000 7f90de74f640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 0x7f90d806fb20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00e9f0 tx=0x7f90cc00edf0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 shutdown_connections 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2cb0 0x7f90c00a5170 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d8102270 0x7f90d8070060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d81018b0 0x7f90d806fb20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 --2- 172.21.15.16:0/3167249527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d81006c0 0x7f90d8073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:45.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 >> 172.21.15.16:0/3167249527 conn(0x7f90d80fbc50 msgr2=0x7f90d80fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 shutdown_connections 2024-04-25T01:31:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:45.987+0000 7f90de74f640 1 -- 172.21.15.16:0/3167249527 wait complete. 2024-04-25T01:31:46.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:31:46.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:31:46.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:45 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:31:46.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:31:46.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:31:46.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:45 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:31:46.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:31:46.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:31:46.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:45 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:31:47.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:46 smithi016 ceph-mon[29153]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:31:47.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:46 smithi038 ceph-mon[32841]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:31:47.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:46 smithi156 ceph-mon[32631]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:31:49.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:48 smithi016 ceph-mon[29153]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-04-25T01:31:49.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:48 smithi038 ceph-mon[32841]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-04-25T01:31:49.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:48 smithi156 ceph-mon[32631]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-04-25T01:31:51.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:50 smithi016 ceph-mon[29153]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:31:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:50 smithi038 ceph-mon[32841]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:31:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:50 smithi156 ceph-mon[32631]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:31:53.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:52 smithi016 ceph-mon[29153]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-04-25T01:31:53.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:52 smithi038 ceph-mon[32841]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-04-25T01:31:53.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:52 smithi156 ceph-mon[32631]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-04-25T01:31:55.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:54 smithi016 ceph-mon[29153]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T01:31:55.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:54 smithi038 ceph-mon[32841]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T01:31:55.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:54 smithi156 ceph-mon[32631]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T01:31:57.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:56 smithi016 ceph-mon[29153]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-25T01:31:57.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:56 smithi038 ceph-mon[32841]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-25T01:31:57.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:56 smithi156 ceph-mon[32631]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-25T01:31:59.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:31:58 smithi016 ceph-mon[29153]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.137+0000 7f6186310640 1 -- 172.21.15.16:0/412096549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6180105580 msgr2=0x7f618010be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.137+0000 7f6186310640 1 --2- 172.21.15.16:0/412096549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6180105580 0x7f618010be10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6178009be0 tx=0x7f617802a6c0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 -- 172.21.15.16:0/412096549 shutdown_connections 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 --2- 172.21.15.16:0/412096549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6180105580 0x7f618010be10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 --2- 172.21.15.16:0/412096549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180104bc0 0x7f6180105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 --2- 172.21.15.16:0/412096549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61801039d0 0x7f6180103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 -- 172.21.15.16:0/412096549 >> 172.21.15.16:0/412096549 conn(0x7f61800fef60 msgr2=0x7f6180101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 -- 172.21.15.16:0/412096549 shutdown_connections 2024-04-25T01:31:59.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.138+0000 7f6186310640 1 -- 172.21.15.16:0/412096549 wait complete. 2024-04-25T01:31:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.139+0000 7f6186310640 1 Processor -- start 2024-04-25T01:31:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 -- start start 2024-04-25T01:31:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61801039d0 0x7f6180199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 0x7f618019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180105580 0x7f61801a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f618010f500 con 0x7f6180105580 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f618010f380 con 0x7f61801039d0 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f6186310640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f618010f680 con 0x7f6180104bc0 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.140+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 0x7f618019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 0x7f618019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:39304/0 (socket says 172.21.15.16:39304) 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 -- 172.21.15.16:0/2515553017 learned_addr learned my addr 172.21.15.16:0/2515553017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617ffff640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61801039d0 0x7f6180199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f61848ad640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180105580 0x7f61801a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.119 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 -- 172.21.15.16:0/2515553017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61801039d0 msgr2=0x7f6180199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61801039d0 0x7f6180199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 -- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180105580 msgr2=0x7f61801a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180105580 0x7f61801a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 -- 172.21.15.16:0/2515553017 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61801a1730 con 0x7f6180104bc0 2024-04-25T01:31:59.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f61848ad640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180105580 0x7f61801a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.141+0000 7f617f7fe640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 0x7f618019a050 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f617400d950 tx=0x7f617400dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.142+0000 7f617d7fa640 1 -- 172.21.15.16:0/2515553017 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174017070 con 0x7f6180104bc0 2024-04-25T01:31:59.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.142+0000 7f617d7fa640 1 -- 172.21.15.16:0/2515553017 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f617400de80 con 0x7f6180104bc0 2024-04-25T01:31:59.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.142+0000 7f617d7fa640 1 -- 172.21.15.16:0/2515553017 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617400f620 con 0x7f6180104bc0 2024-04-25T01:31:59.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.142+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61801a1a20 con 0x7f6180104bc0 2024-04-25T01:31:59.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.142+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61801a1e60 con 0x7f6180104bc0 2024-04-25T01:31:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.145+0000 7f617d7fa640 1 -- 172.21.15.16:0/2515553017 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6174010460 con 0x7f6180104bc0 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 Processor -- start 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 -- start start 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760acb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760af650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a976002cb0 con 0x55a9760a7660 2024-04-25T01:31:59.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a975fab5b0 con 0x55a9760a8050 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.146+0000 7f6189018740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a975ffec70 con 0x55a9760ad1c0 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617d7fa640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61680a2cb0 0x7f61680a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760acb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46430/0 (socket says 172.21.15.16:46430) 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 -- 172.21.15.16:0/2860705395 learned_addr learned my addr 172.21.15.16:0/2860705395 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617f7fe640 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f61848ad640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61680a2cb0 0x7f61680a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f61848ad640 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617d7fa640 1 -- 172.21.15.16:0/2515553017 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f61740ced60 con 0x7f6180104bc0 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 -- 172.21.15.16:0/2860705395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 msgr2=0x55a9760af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760af650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 -- 172.21.15.16:0/2860705395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 msgr2=0x55a9760a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 -- 172.21.15.16:0/2860705395 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a975d17210 con 0x55a9760a8050 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f61848ad640 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.147+0000 7f617ffff640 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760acb50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6170009c80 tx=0x7f6170011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64838cf55f29cd2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.148+0000 7f61667fc640 1 -- 172.21.15.16:0/2860705395 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170004220 con 0x55a9760a8050 2024-04-25T01:31:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.148+0000 7f61667fc640 1 -- 172.21.15.16:0/2860705395 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61700043c0 con 0x55a9760a8050 2024-04-25T01:31:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.148+0000 7f61667fc640 1 -- 172.21.15.16:0/2860705395 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170019690 con 0x55a9760a8050 2024-04-25T01:31:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 -- 172.21.15.16:0/2860705395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 msgr2=0x55a9760acb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f61848ad640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61680a2cb0 0x7f61680a5170 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f617802abe0 tx=0x7f6178009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760acb50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6170009c80 tx=0x7f6170011810 comp rx=0 tx=0).stop 2024-04-25T01:31:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 -- 172.21.15.16:0/2860705395 shutdown_connections 2024-04-25T01:31:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760acb50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 --2- 172.21.15.16:0/2860705395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 -- 172.21.15.16:0/2860705395 >> 172.21.15.16:0/2860705395 conn(0x55a9760a2cd0 msgr2=0x55a9760a30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 -- 172.21.15.16:0/2860705395 shutdown_connections 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.149+0000 7f6189018740 1 -- 172.21.15.16:0/2860705395 wait complete. 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.150+0000 7f6189018740 1 -- start start 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.150+0000 7f6189018740 1 Processor -- start 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.151+0000 7f6189018740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.151+0000 7f617ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.151+0000 7f617ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41000/0 (socket says 172.21.15.16:41000) 2024-04-25T01:31:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.151+0000 7f617ffff640 1 -- 172.21.15.16:0/296334653 learned_addr learned my addr 172.21.15.16:0/296334653 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.151+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.152+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760b9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.152+0000 7f61848ad640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760b9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.152+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a9760b0170 con 0x55a9760a7660 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.152+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a975fbeb50 con 0x55a9760a8050 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.152+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a9760b02f0 con 0x55a9760ad1c0 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.152+0000 7f617f7fe640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617f7fe640 1 -- 172.21.15.16:0/296334653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 msgr2=0x55a9760b9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617f7fe640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760b9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617f7fe640 1 -- 172.21.15.16:0/296334653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 msgr2=0x55a9760b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617f7fe640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.133 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617f7fe640 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a975de3620 con 0x55a9760a8050 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f6189018740 10 client.0 _ll_register_callbacks cb 0x55a975ff9ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61700043c0 con 0x55a9760a8050 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617ffff640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f61848ad640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760b9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f617f7fe640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760b2030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f617409b240 tx=0x7f617401e790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f70fdbec082325c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.153+0000 7f615f7fe640 10 client.25310 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f615f7fe640 1 -- 172.21.15.16:0/296334653 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174020050 con 0x55a9760a8050 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f615f7fe640 1 -- 172.21.15.16:0/296334653 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6174097bb0 con 0x55a9760a8050 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f615f7fe640 1 -- 172.21.15.16:0/296334653 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f617409aea0 con 0x55a9760a8050 2024-04-25T01:31:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f6189018740 10 client.25310 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55a9760bc340 con 0x55a9760a8050 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f615f7fe640 1 -- 172.21.15.16:0/296334653 <== mon.2 v2:172.21.15.38:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6174066180 con 0x55a9760a8050 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f6189018740 10 client.25310 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f6189018740 10 client.25310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f6189018740 20 client.25310 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.154+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a9760bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.155+0000 7f6189018740 4 client.25310 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.155+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9760101f0 con 0x55a9760bd9f0 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.155+0000 7f617ffff640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a9760bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.155+0000 7f615f7fe640 10 client.25310 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:59.135 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.155+0000 7f617ffff640 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a9760bfde0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37182d52cc2f4ece server_cookie=ea06c0ba1a96a5f4 in_seq=0 out_seq=0 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f615f7fe640 1 -- 172.21.15.16:0/296334653 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9760101f0 con 0x55a9760bd9f0 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f615f7fe640 10 client.25310 handle_command_reply: tid=0 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 client.25310 shutdown 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 msgr2=0x55a9760b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760b2030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f617409b240 tx=0x7f617401e790 comp rx=0 tx=0).stop 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 shutdown_connections 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760ad1c0 0x55a9760b9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a9760bfde0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760a8050 0x55a9760b2030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 --2- 172.21.15.16:0/296334653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a7660 0x55a9760b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f615f7fe640 0 client.25310 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.156+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 >> 172.21.15.16:0/296334653 conn(0x55a9760a2cd0 msgr2=0x55a9760995c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.157+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 shutdown_connections 2024-04-25T01:31:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.157+0000 7f6189018740 1 -- 172.21.15.16:0/296334653 wait complete. 2024-04-25T01:31:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.157+0000 7f6189018740 20 client.25310 trim_cache size 0 max 16384 2024-04-25T01:31:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 Processor -- start 2024-04-25T01:31:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 -- start start 2024-04-25T01:31:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a976098bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760990f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61801a9320 con 0x55a9760a8050 2024-04-25T01:31:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f61801a6bf0 con 0x55a9760ad1c0 2024-04-25T01:31:59.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.186+0000 7f6189018740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f61801a4a60 con 0x55a9760a7660 2024-04-25T01:31:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46446/0 (socket says 172.21.15.16:46446) 2024-04-25T01:31:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 -- 172.21.15.16:0/910503953 learned_addr learned my addr 172.21.15.16:0/910503953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f617ffff640 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a976098bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f617f7fe640 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 -- 172.21.15.16:0/910503953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 msgr2=0x55a976098bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a976098bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 -- 172.21.15.16:0/910503953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 msgr2=0x55a9760b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 -- 172.21.15.16:0/910503953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9760c0150 con 0x55a9760ad1c0 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.187+0000 7f61848ad640 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760990f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6178009be0 tx=0x7f617803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ddabb1f8f20835 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.188+0000 7f6165ffb640 1 -- 172.21.15.16:0/910503953 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617804e070 con 0x55a9760ad1c0 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.188+0000 7f6165ffb640 1 -- 172.21.15.16:0/910503953 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6178043040 con 0x55a9760ad1c0 2024-04-25T01:31:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.188+0000 7f6165ffb640 1 -- 172.21.15.16:0/910503953 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178036040 con 0x55a9760ad1c0 2024-04-25T01:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 -- 172.21.15.16:0/910503953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 msgr2=0x55a9760990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760990f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6178009be0 tx=0x7f617803d000 comp rx=0 tx=0).stop 2024-04-25T01:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 -- 172.21.15.16:0/910503953 shutdown_connections 2024-04-25T01:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760990f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.169 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 --2- 172.21.15.16:0/910503953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a976098bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 -- 172.21.15.16:0/910503953 >> 172.21.15.16:0/910503953 conn(0x55a9760a2cd0 msgr2=0x55a9760a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 -- 172.21.15.16:0/910503953 shutdown_connections 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.189+0000 7f6189018740 1 -- 172.21.15.16:0/910503953 wait complete. 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 -- start start 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 Processor -- start 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a9760b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760b7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61801a50b0 con 0x55a9760a8050 2024-04-25T01:31:59.170 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f61801a6520 con 0x55a9760ad1c0 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f61801a5780 con 0x55a9760a7660 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.190+0000 7f6189018740 10 client.0 _ll_register_callbacks cb 0x7f6174009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f61848ad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41006/0 (socket says 172.21.15.16:41006) 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 -- 172.21.15.16:0/403602114 learned_addr learned my addr 172.21.15.16:0/403602114 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f61848ad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760b7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46448/0 (socket says 172.21.15.16:46448) 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617ffff640 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a9760b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 -- 172.21.15.16:0/403602114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 msgr2=0x55a9760b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a9760b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 -- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 msgr2=0x55a9760b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 -- 172.21.15.16:0/403602114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9760b7f00 con 0x55a9760a8050 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f617f7fe640 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b6900 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f617401d040 tx=0x7f617406a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d712ecdcdac4795 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f615f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f615f7fe640 1 -- 172.21.15.16:0/403602114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174095060 con 0x55a9760a8050 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.191+0000 7f615f7fe640 1 -- 172.21.15.16:0/403602114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f617409b9b0 con 0x55a9760a8050 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6178043040 con 0x55a9760a8050 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f615f7fe640 1 -- 172.21.15.16:0/403602114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f617409b310 con 0x55a9760a8050 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 10 client.15630 fetch_fsmap learned FSMap version 22 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a9760b9210 con 0x55a9760a8050 2024-04-25T01:31:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f615f7fe640 1 -- 172.21.15.16:0/403602114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61700043c0 con 0x55a9760a8050 2024-04-25T01:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f615f7fe640 1 -- 172.21.15.16:0/403602114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6174010750 con 0x55a9760a8050 2024-04-25T01:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 10 client.15630 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 20 client.15630 populate_metadata read hostname 'smithi016' 2024-04-25T01:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a976126490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 4 client.15630 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f6178043040 con 0x55a9760bd9f0 2024-04-25T01:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.192+0000 7f617ffff640 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a976126490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.193+0000 7f617ffff640 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a976126490 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad852ed24f77ff87 server_cookie=72f36e8e4bf29ed9 in_seq=0 out_seq=0 2024-04-25T01:31:59.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.193+0000 7f615f7fe640 10 client.15630 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f615f7fe640 1 -- 172.21.15.16:0/403602114 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f6178043040 con 0x55a9760bd9f0 2024-04-25T01:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f615f7fe640 10 client.15630 handle_command_reply: tid=0 2024-04-25T01:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 client.15630 shutdown 2024-04-25T01:31:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 msgr2=0x55a9760b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b6900 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f617401d040 tx=0x7f617406a4c0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 shutdown_connections 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55a9760bd9f0 0x55a976126490 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9760ad1c0 0x55a9760b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a9760a8050 0x55a9760b6900 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 --2- 172.21.15.16:0/403602114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a9760a7660 0x55a9760b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 >> 172.21.15.16:0/403602114 conn(0x55a9760a2cd0 msgr2=0x55a9760a5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f615f7fe640 0 client.15630 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:31:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 shutdown_connections 2024-04-25T01:31:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.197+0000 7f6189018740 1 -- 172.21.15.16:0/403602114 wait complete. 2024-04-25T01:31:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.198+0000 7f6189018740 20 client.15630 trim_cache size 0 max 16384 2024-04-25T01:31:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:31:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:31:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:31:59.177 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:31:59.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.198+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61680a2cb0 msgr2=0x7f61680a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.198+0000 7f6186310640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61680a2cb0 0x7f61680a5170 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f617802abe0 tx=0x7f6178009500 comp rx=0 tx=0).stop 2024-04-25T01:31:59.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.198+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 msgr2=0x7f618019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.198+0000 7f6186310640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 0x7f618019a050 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f617400d950 tx=0x7f617400dd50 comp rx=0 tx=0).stop 2024-04-25T01:31:59.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 shutdown_connections 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61680a2cb0 0x7f61680a5170 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6180105580 0x7f61801a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6180104bc0 0x7f618019a050 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 --2- 172.21.15.16:0/2515553017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61801039d0 0x7f6180199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 >> 172.21.15.16:0/2515553017 conn(0x7f61800fef60 msgr2=0x7f6180107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 shutdown_connections 2024-04-25T01:31:59.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:31:59.199+0000 7f6186310640 1 -- 172.21.15.16:0/2515553017 wait complete. 2024-04-25T01:31:59.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:31:58 smithi038 ceph-mon[32841]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:31:59.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:31:58 smithi156 ceph-mon[32631]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:32:01.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:00 smithi016 ceph-mon[29153]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:01.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:00 smithi038 ceph-mon[32841]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:01.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:00 smithi156 ceph-mon[32631]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:03.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:02 smithi016 ceph-mon[29153]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:03.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:02 smithi038 ceph-mon[32841]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:03.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:02 smithi156 ceph-mon[32631]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:05.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:04 smithi016 ceph-mon[29153]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:32:05.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:04 smithi038 ceph-mon[32841]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:32:05.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:04 smithi156 ceph-mon[32631]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-25T01:32:07.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:06 smithi016 ceph-mon[29153]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:32:07.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:06 smithi038 ceph-mon[32841]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:32:07.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:06 smithi156 ceph-mon[32631]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:32:09.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:08 smithi016 ceph-mon[29153]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:09.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:08 smithi038 ceph-mon[32841]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:09.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:08 smithi156 ceph-mon[32631]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:11.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:10 smithi016 ceph-mon[29153]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:11.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:10 smithi038 ceph-mon[32841]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:11.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:10 smithi156 ceph-mon[32631]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:12.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f0165d9b640 1 --2- 172.21.15.16:0/2031835527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 0x7f0168100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 -- 172.21.15.16:0/2031835527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0168101190 msgr2=0x7f0168101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 --2- 172.21.15.16:0/2031835527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0168101190 0x7f0168101610 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0160009ca0 tx=0x7f016002a8d0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 -- 172.21.15.16:0/2031835527 shutdown_connections 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 --2- 172.21.15.16:0/2031835527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0168101190 0x7f0168101610 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 --2- 172.21.15.16:0/2031835527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 0x7f0168100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 --2- 172.21.15.16:0/2031835527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01680ffe90 0x7f01681002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 -- 172.21.15.16:0/2031835527 >> 172.21.15.16:0/2031835527 conn(0x7f01680fba70 msgr2=0x7f01680fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.348+0000 7f016759e640 1 -- 172.21.15.16:0/2031835527 shutdown_connections 2024-04-25T01:32:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.349+0000 7f016759e640 1 -- 172.21.15.16:0/2031835527 wait complete. 2024-04-25T01:32:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 Processor -- start 2024-04-25T01:32:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 -- start start 2024-04-25T01:32:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01680ffe90 0x7f0168073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 0x7f016806fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 0x7f01680703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f016810f400 con 0x7f01680ffe90 2024-04-25T01:32:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f016810f280 con 0x7f0168101190 2024-04-25T01:32:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016759e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f016810f580 con 0x7f01681007f0 2024-04-25T01:32:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01680ffe90 0x7f0168073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f0166d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 0x7f01680703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01680ffe90 0x7f0168073390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40920/0 (socket says 172.21.15.16:40920) 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f0166d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 0x7f01680703c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54894/0 (socket says 172.21.15.16:54894) 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.350+0000 7f016659c640 1 -- 172.21.15.16:0/1534359018 learned_addr learned my addr 172.21.15.16:0/1534359018 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0165d9b640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 0x7f016806fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0166d9d640 1 -- 172.21.15.16:0/1534359018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 msgr2=0x7f016806fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0166d9d640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 0x7f016806fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0166d9d640 1 -- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01680ffe90 msgr2=0x7f0168073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0166d9d640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01680ffe90 0x7f0168073390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0166d9d640 1 -- 172.21.15.16:0/1534359018 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01681a9bc0 con 0x7f0168101190 2024-04-25T01:32:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.351+0000 7f0166d9d640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 0x7f01680703c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f016002ae40 tx=0x7f0160003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.352+0000 7f01577fe640 1 -- 172.21.15.16:0/1534359018 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160004240 con 0x7f0168101190 2024-04-25T01:32:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.352+0000 7f01577fe640 1 -- 172.21.15.16:0/1534359018 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01600305d0 con 0x7f0168101190 2024-04-25T01:32:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.352+0000 7f01577fe640 1 -- 172.21.15.16:0/1534359018 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016003c650 con 0x7f0168101190 2024-04-25T01:32:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.352+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01681a9e50 con 0x7f0168101190 2024-04-25T01:32:12.332 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.352+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01681aa340 con 0x7f0168101190 2024-04-25T01:32:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.355+0000 7f016fd74740 1 Processor -- start 2024-04-25T01:32:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.356+0000 7f01577fe640 1 -- 172.21.15.16:0/1534359018 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0160009780 con 0x7f0168101190 2024-04-25T01:32:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.356+0000 7f016fd74740 1 -- start start 2024-04-25T01:32:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f016fd74740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f42f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f016fd74740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f434520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f016fd74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f437020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f016fd74740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55572f392cb0 con 0x55572f42fa20 2024-04-25T01:32:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f016fd74740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55572f33b5b0 con 0x55572f42f030 2024-04-25T01:32:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f016fd74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55572f38ec70 con 0x55572f434b90 2024-04-25T01:32:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f01577fe640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f01500a2cc0 0x7f01500a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f01577fe640 1 -- 172.21.15.16:0/1534359018 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f01600f0800 con 0x7f0168101190 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f016659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f434520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.357+0000 7f0165d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f42f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f016659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f434520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40928/0 (socket says 172.21.15.16:40928) 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f016659c640 1 -- 172.21.15.16:0/167262178 learned_addr learned my addr 172.21.15.16:0/167262178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0165d9b640 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f437020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0166d9d640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f01500a2cc0 0x7f01500a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0165d9b640 1 -- 172.21.15.16:0/167262178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 msgr2=0x55572f437020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0165d9b640 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f437020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0165d9b640 1 -- 172.21.15.16:0/167262178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 msgr2=0x55572f434520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0165d9b640 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f434520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.358+0000 7f0165d9b640 1 -- 172.21.15.16:0/167262178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55572f0a7210 con 0x55572f42f030 2024-04-25T01:32:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.359+0000 7f0165d9b640 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f42f450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0168071310 tx=0x7f015c00a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca58bb94e5a6d81e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.359+0000 7f0166d9d640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f01500a2cc0 0x7f01500a5180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f01600b9440 tx=0x7f0160032f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.359+0000 7f0154ff9640 1 -- 172.21.15.16:0/167262178 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c01c310 con 0x55572f42f030 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.359+0000 7f0154ff9640 1 -- 172.21.15.16:0/167262178 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f015c00fe60 con 0x55572f42f030 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.359+0000 7f0154ff9640 1 -- 172.21.15.16:0/167262178 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c005280 con 0x55572f42f030 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.360+0000 7f016fd74740 1 -- 172.21.15.16:0/167262178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 msgr2=0x55572f42f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.360+0000 7f016fd74740 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f42f450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0168071310 tx=0x7f015c00a6c0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.360+0000 7f016fd74740 1 -- 172.21.15.16:0/167262178 shutdown_connections 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.360+0000 7f016fd74740 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f437020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.360+0000 7f016fd74740 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f434520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.344 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.361+0000 7f016fd74740 1 --2- 172.21.15.16:0/167262178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f42f450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.361+0000 7f016fd74740 1 -- 172.21.15.16:0/167262178 >> 172.21.15.16:0/167262178 conn(0x55572f42a6a0 msgr2=0x55572f42aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.361+0000 7f016fd74740 1 -- 172.21.15.16:0/167262178 shutdown_connections 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.361+0000 7f016fd74740 1 -- 172.21.15.16:0/167262178 wait complete. 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.361+0000 7f016fd74740 1 -- start start 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.361+0000 7f016fd74740 1 Processor -- start 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42f030 0x55572f4395a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 0x55572f439ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f440ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55572f437b40 con 0x55572f42f030 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55572f34eb50 con 0x55572f42fa20 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55572f437cc0 con 0x55572f434b90 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 0x55572f439ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 0x55572f439ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54916/0 (socket says 172.21.15.16:54916) 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 -- 172.21.15.16:0/3935835704 learned_addr learned my addr 172.21.15.16:0/3935835704 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016fd74740 10 client.0 _ll_register_callbacks cb 0x55572f389ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42f030 0x55572f4395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 -- 172.21.15.16:0/3935835704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 msgr2=0x55572f440ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f440ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 -- 172.21.15.16:0/3935835704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42f030 msgr2=0x55572f4395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42f030 0x55572f4395a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f016659c640 1 -- 172.21.15.16:0/3935835704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572f173620 con 0x55572f42fa20 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.362+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42f030 0x55572f4395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f016659c640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 0x55572f439ae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f01580115b0 tx=0x7f0158011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4be8956ff87aaa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f014dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f014dffb640 1 -- 172.21.15.16:0/3935835704 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015800f6f0 con 0x55572f42fa20 2024-04-25T01:32:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015c00fe60 con 0x55572f42fa20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f014dffb640 1 -- 172.21.15.16:0/3935835704 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f015800f890 con 0x55572f42fa20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f014dffb640 1 -- 172.21.15.16:0/3935835704 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0158023070 con 0x55572f42fa20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f016fd74740 10 client.25325 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55572f443ec0 con 0x55572f42fa20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.363+0000 7f014dffb640 1 -- 172.21.15.16:0/3935835704 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f015c00fe60 con 0x55572f42fa20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f014dffb640 1 -- 172.21.15.16:0/3935835704 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0158014bf0 con 0x55572f42fa20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f016fd74740 10 client.25325 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f016fd74740 10 client.25325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f016fd74740 20 client.25325 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f016fd74740 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f445a20 0x55572f447ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f016fd74740 4 client.25325 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55572f3a01f0 con 0x55572f445a20 2024-04-25T01:32:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.364+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f445a20 0x55572f447ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.365+0000 7f014dffb640 10 client.25325 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.365+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f445a20 0x55572f447ea0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=973d6654231f389b server_cookie=ff24ab108426fdc4 in_seq=0 out_seq=0 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.365+0000 7f014dffb640 1 -- 172.21.15.16:0/3935835704 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55572f3a01f0 con 0x55572f445a20 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.365+0000 7f014dffb640 10 client.25325 handle_command_reply: tid=0 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.365+0000 7f016fd74740 1 client.25325 shutdown 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 msgr2=0x55572f439ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 0x55572f439ae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f01580115b0 tx=0x7f0158011a70 comp rx=0 tx=0).stop 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 shutdown_connections 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f440ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f445a20 0x55572f447ea0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42fa20 0x55572f439ae0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 --2- 172.21.15.16:0/3935835704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42f030 0x55572f4395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f014dffb640 0 client.25325 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 >> 172.21.15.16:0/3935835704 conn(0x55572f42a6a0 msgr2=0x55572f4298a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 shutdown_connections 2024-04-25T01:32:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 1 -- 172.21.15.16:0/3935835704 wait complete. 2024-04-25T01:32:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.366+0000 7f016fd74740 20 client.25325 trim_cache size 0 max 16384 2024-04-25T01:32:12.372 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.394+0000 7f016fd74740 1 Processor -- start 2024-04-25T01:32:12.372 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.394+0000 7f016fd74740 1 -- start start 2024-04-25T01:32:12.372 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016fd74740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f429010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.372 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016fd74740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f440a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016fd74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f429550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016fd74740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b1720 con 0x55572f42fa20 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016fd74740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f01681aeff0 con 0x55572f42f030 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016fd74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ace40 con 0x55572f434b90 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f440a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f440a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40940/0 (socket says 172.21.15.16:40940) 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 -- 172.21.15.16:0/1538510506 learned_addr learned my addr 172.21.15.16:0/1538510506 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f0165d9b640 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f429010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f0165d9b640 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f429550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 -- 172.21.15.16:0/1538510506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 msgr2=0x55572f429550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f429550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 -- 172.21.15.16:0/1538510506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 msgr2=0x55572f429010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f429010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f016659c640 1 -- 172.21.15.16:0/1538510506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55572f448520 con 0x55572f42fa20 2024-04-25T01:32:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.395+0000 7f0165d9b640 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f429010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.396+0000 7f016659c640 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f440a70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0158006d90 tx=0x7f0158000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e47fd7b7787d5b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.396+0000 7f014ffff640 1 -- 172.21.15.16:0/1538510506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015802c020 con 0x55572f42fa20 2024-04-25T01:32:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.396+0000 7f014ffff640 1 -- 172.21.15.16:0/1538510506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f015801dd40 con 0x55572f42fa20 2024-04-25T01:32:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.396+0000 7f014ffff640 1 -- 172.21.15.16:0/1538510506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158028940 con 0x55572f42fa20 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.397+0000 7f016fd74740 1 -- 172.21.15.16:0/1538510506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 msgr2=0x55572f440a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.397+0000 7f016fd74740 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f440a70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0158006d90 tx=0x7f0158000bc0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 -- 172.21.15.16:0/1538510506 shutdown_connections 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f434b90 0x55572f429550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f42fa20 0x55572f440a70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 --2- 172.21.15.16:0/1538510506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f429010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 -- 172.21.15.16:0/1538510506 >> 172.21.15.16:0/1538510506 conn(0x55572f42a6a0 msgr2=0x55572f4395a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 -- 172.21.15.16:0/1538510506 shutdown_connections 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 -- 172.21.15.16:0/1538510506 wait complete. 2024-04-25T01:32:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.398+0000 7f016fd74740 1 -- start start 2024-04-25T01:32:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 Processor -- start 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f4408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 0x55572f43d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f434b90 0x55572f43d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ad4b0 con 0x55572f434b90 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ae920 con 0x55572f42f030 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f01681adb80 con 0x55572f42fa20 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016fd74740 10 client.0 _ll_register_callbacks cb 0x7f0158009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 0x55572f43d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 0x55572f43d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:33292/0 (socket says 172.21.15.16:33292) 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f016659c640 1 -- 172.21.15.16:0/3931974452 learned_addr learned my addr 172.21.15.16:0/3931974452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.399+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f434b90 0x55572f43d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016659c640 1 -- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 msgr2=0x55572f4408d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016659c640 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f4408d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016659c640 1 -- 172.21.15.16:0/3931974452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f434b90 msgr2=0x55572f43d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016659c640 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f434b90 0x55572f43d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016659c640 1 -- 172.21.15.16:0/3931974452 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572f4aa010 con 0x55572f42fa20 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016659c640 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 0x55572f43d370 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f015801d5c0 tx=0x7f015800f920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a3d8f3bb6178853 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f014dffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f014dffb640 1 -- 172.21.15.16:0/3931974452 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158035070 con 0x55572f42fa20 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f014dffb640 1 -- 172.21.15.16:0/3931974452 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f015800fca0 con 0x55572f42fa20 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.400+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015801dd40 con 0x55572f42fa20 2024-04-25T01:32:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f014dffb640 1 -- 172.21.15.16:0/3931974452 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f015803f050 con 0x55572f42fa20 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 10 client.25297 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55572f4ab170 con 0x55572f42fa20 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f014dffb640 1 -- 172.21.15.16:0/3931974452 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f015801dd40 con 0x55572f42fa20 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f014dffb640 1 -- 172.21.15.16:0/3931974452 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f01580289c0 con 0x55572f42fa20 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 10 client.25297 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 10 client.25297 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 20 client.25297 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f4abb40 0x55572f4adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 4 client.25297 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55572f39a540 con 0x55572f4abb40 2024-04-25T01:32:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.401+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f4abb40 0x55572f4adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.402+0000 7f0165d9b640 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f4abb40 0x55572f4adf60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af98e79b80ba46c2 server_cookie=66e743581651313c in_seq=0 out_seq=0 2024-04-25T01:32:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.402+0000 7f014dffb640 10 client.25297 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f014dffb640 1 -- 172.21.15.16:0/3931974452 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55572f39a540 con 0x55572f4abb40 2024-04-25T01:32:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f014dffb640 10 client.25297 handle_command_reply: tid=0 2024-04-25T01:32:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 client.25297 shutdown 2024-04-25T01:32:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 msgr2=0x55572f43d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 0x55572f43d370 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f015801d5c0 tx=0x7f015800f920 comp rx=0 tx=0).stop 2024-04-25T01:32:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 shutdown_connections 2024-04-25T01:32:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55572f434b90 0x55572f43d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55572f42fa20 0x55572f43d370 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55572f4abb40 0x55572f4adf60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.406+0000 7f016fd74740 1 --2- 172.21.15.16:0/3931974452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55572f42f030 0x55572f4408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.407+0000 7f014dffb640 0 client.25297 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.407+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 >> 172.21.15.16:0/3931974452 conn(0x55572f42a6a0 msgr2=0x55572f438090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.407+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 shutdown_connections 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.407+0000 7f016fd74740 1 -- 172.21.15.16:0/3931974452 wait complete. 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.407+0000 7f016fd74740 20 client.25297 trim_cache size 0 max 16384 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:32:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:32:12.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f01500a2cc0 msgr2=0x7f01500a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f01500a2cc0 0x7f01500a5180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f01600b9440 tx=0x7f0160032f90 comp rx=0 tx=0).stop 2024-04-25T01:32:12.386 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 msgr2=0x7f01680703c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 0x7f01680703c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f016002ae40 tx=0x7f0160003f10 comp rx=0 tx=0).stop 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 shutdown_connections 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f01500a2cc0 0x7f01500a5180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0168101190 0x7f01680703c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01681007f0 0x7f016806fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 --2- 172.21.15.16:0/1534359018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01680ffe90 0x7f0168073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 >> 172.21.15.16:0/1534359018 conn(0x7f01680fba70 msgr2=0x7f01680fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 shutdown_connections 2024-04-25T01:32:12.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:12.408+0000 7f016759e640 1 -- 172.21.15.16:0/1534359018 wait complete. 2024-04-25T01:32:13.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:12 smithi016 ceph-mon[29153]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:13.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:12 smithi038 ceph-mon[32841]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:13.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:12 smithi156 ceph-mon[32631]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:15.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:14 smithi016 ceph-mon[29153]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:32:15.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:14 smithi038 ceph-mon[32841]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:32:15.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:14 smithi156 ceph-mon[32631]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-25T01:32:17.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:16 smithi016 ceph-mon[29153]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-04-25T01:32:17.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:16 smithi038 ceph-mon[32841]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-04-25T01:32:17.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:16 smithi156 ceph-mon[32631]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-04-25T01:32:19.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:18 smithi016 ceph-mon[29153]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:19.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:18 smithi038 ceph-mon[32841]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:19.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:18 smithi156 ceph-mon[32631]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:21.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:20 smithi016 ceph-mon[29153]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:21.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:20 smithi038 ceph-mon[32841]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:21.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:20 smithi156 ceph-mon[32631]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:23.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:22 smithi016 ceph-mon[29153]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:32:23.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:22 smithi038 ceph-mon[32841]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:32:23.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:22 smithi156 ceph-mon[32631]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:32:25.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:24 smithi038 ceph-mon[32841]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:25.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:24 smithi156 ceph-mon[32631]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:25.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:24 smithi016 ceph-mon[29153]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:32:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 -- 172.21.15.16:0/151374291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c010dba0 msgr2=0x7fa9c010ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 --2- 172.21.15.16:0/151374291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c010dba0 0x7fa9c010ff90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009cd0 tx=0x7fa9b802a8f0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 -- 172.21.15.16:0/151374291 shutdown_connections 2024-04-25T01:32:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 --2- 172.21.15.16:0/151374291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c010dba0 0x7fa9c010ff90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 --2- 172.21.15.16:0/151374291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c00741a0 0x7fa9c0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 --2- 172.21.15.16:0/151374291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9c0075780 0x7fa9c0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 -- 172.21.15.16:0/151374291 >> 172.21.15.16:0/151374291 conn(0x7fa9c00fcb40 msgr2=0x7fa9c00fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 -- 172.21.15.16:0/151374291 shutdown_connections 2024-04-25T01:32:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.837+0000 7fa9c503c640 1 -- 172.21.15.16:0/151374291 wait complete. 2024-04-25T01:32:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.839+0000 7fa9c503c640 1 Processor -- start 2024-04-25T01:32:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.839+0000 7fa9c503c640 1 -- start start 2024-04-25T01:32:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.839+0000 7fa9c503c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 0x7fa9c019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.839+0000 7fa9c503c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c0075780 0x7fa9c019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 0x7fa9c019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 0x7fa9c019dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43270/0 (socket says 172.21.15.16:43270) 2024-04-25T01:32:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9bffff640 1 -- 172.21.15.16:0/2788568848 learned_addr learned my addr 172.21.15.16:0/2788568848 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9c010dba0 0x7fa9c01a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c0075780 0x7fa9c019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0113720 con 0x7fa9c0075780 2024-04-25T01:32:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01135a0 con 0x7fa9c00741a0 2024-04-25T01:32:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01138a0 con 0x7fa9c010dba0 2024-04-25T01:32:25.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.840+0000 7fa9c483b640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9c010dba0 0x7fa9c01a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9b7fff640 1 -- 172.21.15.16:0/2788568848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9c010dba0 msgr2=0x7fa9c01a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9c010dba0 0x7fa9c01a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9b7fff640 1 -- 172.21.15.16:0/2788568848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 msgr2=0x7fa9c019dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 0x7fa9c019dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9b7fff640 1 -- 172.21.15.16:0/2788568848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c01a5990 con 0x7fa9c0075780 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9bffff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 0x7fa9c019dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c0075780 0x7fa9c019e2b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800e9f0 tx=0x7fa9a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9bdffb640 1 -- 172.21.15.16:0/2788568848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a800cd80 con 0x7fa9c0075780 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c01a5c80 con 0x7fa9c0075780 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9bdffb640 1 -- 172.21.15.16:0/2788568848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9a80035f0 con 0x7fa9c0075780 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.841+0000 7fa9bdffb640 1 -- 172.21.15.16:0/2788568848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a800cd80 con 0x7fa9c0075780 2024-04-25T01:32:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.842+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c01a61d0 con 0x7fa9c0075780 2024-04-25T01:32:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.844+0000 7fa9bdffb640 1 -- 172.21.15.16:0/2788568848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa9a80037a0 con 0x7fa9c0075780 2024-04-25T01:32:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.845+0000 7fa9bdffb640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa9ac0a2d80 0x7fa9ac0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.846+0000 7fa9bdffb640 1 -- 172.21.15.16:0/2788568848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa9a80cef30 con 0x7fa9c0075780 2024-04-25T01:32:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.846+0000 7fa9bffff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa9ac0a2d80 0x7fa9ac0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.847+0000 7fa9bffff640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa9ac0a2d80 0x7fa9ac0a5240 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000f4d0 tx=0x7fa9b000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 Processor -- start 2024-04-25T01:32:25.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 -- start start 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2030 0x55c0210d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 0x55c0210d7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210da020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c021035cb0 con 0x55c0210d7b90 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c020fde5b0 con 0x55c0210d2a20 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.880+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c021031c70 con 0x55c0210d2030 2024-04-25T01:32:25.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.881+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 0x55c0210d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.881+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 0x55c0210d7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43278/0 (socket says 172.21.15.16:43278) 2024-04-25T01:32:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.881+0000 7fa9b7fff640 1 -- 172.21.15.16:0/2836055599 learned_addr learned my addr 172.21.15.16:0/2836055599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.881+0000 7fa9c483b640 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210da020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9c483b640 1 -- 172.21.15.16:0/2836055599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2030 msgr2=0x55c0210d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9c483b640 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2030 0x55c0210d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9c483b640 1 -- 172.21.15.16:0/2836055599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 msgr2=0x55c0210d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9c483b640 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 0x55c0210d7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9c483b640 1 -- 172.21.15.16:0/2836055599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c020d4a210 con 0x55c0210d7b90 2024-04-25T01:32:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 0x55c0210d7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.882+0000 7fa9c483b640 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210da020 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa9b802ae10 tx=0x7fa9b8009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b221252c779380 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.885+0000 7fa9b67fc640 1 -- 172.21.15.16:0/2836055599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8030960 con 0x55c0210d7b90 2024-04-25T01:32:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.885+0000 7fa9b67fc640 1 -- 172.21.15.16:0/2836055599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b8036030 con 0x55c0210d7b90 2024-04-25T01:32:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.885+0000 7fa9b67fc640 1 -- 172.21.15.16:0/2836055599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8004510 con 0x55c0210d7b90 2024-04-25T01:32:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 -- 172.21.15.16:0/2836055599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 msgr2=0x55c0210da020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210da020 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa9b802ae10 tx=0x7fa9b8009a20 comp rx=0 tx=0).stop 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 -- 172.21.15.16:0/2836055599 shutdown_connections 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210da020 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2a20 0x55c0210d7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/2836055599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2030 0x55c0210d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 -- 172.21.15.16:0/2836055599 >> 172.21.15.16:0/2836055599 conn(0x55c0210cd6a0 msgr2=0x55c0210cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 -- 172.21.15.16:0/2836055599 shutdown_connections 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.886+0000 7fa9c8fad740 1 -- 172.21.15.16:0/2836055599 wait complete. 2024-04-25T01:32:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.887+0000 7fa9c8fad740 1 -- start start 2024-04-25T01:32:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.887+0000 7fa9c8fad740 1 Processor -- start 2024-04-25T01:32:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.888+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2030 0x55c0210dc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.888+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210dc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.888+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2030 0x55c0210dc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.888+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2030 0x55c0210dc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43282/0 (socket says 172.21.15.16:43282) 2024-04-25T01:32:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.888+0000 7fa9bffff640 1 -- 172.21.15.16:0/716753626 learned_addr learned my addr 172.21.15.16:0/716753626 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.888+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210dc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.889+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.889+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0210daae0 con 0x55c0210d7b90 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.889+0000 7fa9c483b640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.889+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c020ff1b50 con 0x55c0210d2030 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.889+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c0210dac60 con 0x55c0210d2a20 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9c8fad740 10 client.0 _ll_register_callbacks cb 0x55c02102cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9b7fff640 1 -- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2030 msgr2=0x55c0210dc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2030 0x55c0210dc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9b7fff640 1 -- 172.21.15.16:0/716753626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 msgr2=0x55c0210e3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210e3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9b7fff640 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c020e16620 con 0x55c0210d2a20 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210dc9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9a809b6a0 tx=0x7fa9a809b160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ed70b215500173 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa99f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa99f7fe640 1 -- 172.21.15.16:0/716753626 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80b3020 con 0x55c0210d2a20 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa99f7fe640 1 -- 172.21.15.16:0/716753626 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa9a8009390 con 0x55c0210d2a20 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.890+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9b8036030 con 0x55c0210d2a20 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.891+0000 7fa99f7fe640 1 -- 172.21.15.16:0/716753626 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa9a809fac0 con 0x55c0210d2a20 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.891+0000 7fa9c8fad740 10 client.25303 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.891+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c0210e6df0 con 0x55c0210d2a20 2024-04-25T01:32:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.891+0000 7fa99f7fe640 1 -- 172.21.15.16:0/716753626 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b8036030 con 0x55c0210d2a20 2024-04-25T01:32:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.891+0000 7fa99f7fe640 1 -- 172.21.15.16:0/716753626 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa9a80974f0 con 0x55c0210d2a20 2024-04-25T01:32:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9c8fad740 10 client.25303 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9c8fad740 10 client.25303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9c8fad740 20 client.25303 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0210e8530 0x55c0210ea960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9c8fad740 4 client.25303 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0210431f0 con 0x55c0210e8530 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.892+0000 7fa9bffff640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0210e8530 0x55c0210ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.893+0000 7fa99f7fe640 10 client.25303 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.893+0000 7fa9bffff640 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0210e8530 0x55c0210ea960 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2d97dd5de455472 server_cookie=3ebb81d095a23d10 in_seq=0 out_seq=0 2024-04-25T01:32:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa99f7fe640 1 -- 172.21.15.16:0/716753626 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0210431f0 con 0x55c0210e8530 2024-04-25T01:32:25.872 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa99f7fe640 10 client.25303 handle_command_reply: tid=0 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa9c8fad740 1 client.25303 shutdown 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 msgr2=0x55c0210dc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210dc9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9a809b6a0 tx=0x7fa9a809b160 comp rx=0 tx=0).stop 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 shutdown_connections 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.894+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d7b90 0x55c0210e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210dc9c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c0210e8530 0x55c0210ea960 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/716753626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d2030 0x55c0210dc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 >> 172.21.15.16:0/716753626 conn(0x55c0210cd6a0 msgr2=0x55c0210cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa99f7fe640 0 client.25303 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 shutdown_connections 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 1 -- 172.21.15.16:0/716753626 wait complete. 2024-04-25T01:32:25.873 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.895+0000 7fa9c8fad740 20 client.25303 trim_cache size 0 max 16384 2024-04-25T01:32:25.897 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.920+0000 7fa9c8fad740 1 Processor -- start 2024-04-25T01:32:25.898 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.920+0000 7fa9c8fad740 1 -- start start 2024-04-25T01:32:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.920+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c0210d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.902 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01b18a0 con 0x55c0210d2030 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01af170 con 0x55c0210d7b90 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01acfc0 con 0x55c0210d2a20 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51338/0 (socket says 172.21.15.16:51338) 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:55526/0 (socket says 172.21.15.16:55526) 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9bffff640 1 -- 172.21.15.16:0/1996545269 learned_addr learned my addr 172.21.15.16:0/1996545269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c483b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c0210d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c483b640 1 -- 172.21.15.16:0/1996545269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 msgr2=0x55c0210e39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c483b640 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c483b640 1 -- 172.21.15.16:0/1996545269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 msgr2=0x55c0210d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c483b640 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.921+0000 7fa9c483b640 1 -- 172.21.15.16:0/1996545269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0210ce410 con 0x55c0210d7b90 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.922+0000 7fa9c483b640 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c0210d0b90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa9b802aee0 tx=0x7fa9b802a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=566d0c6b48e1f8d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.922+0000 7fa9b5ffb640 1 -- 172.21.15.16:0/1996545269 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80304e0 con 0x55c0210d7b90 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.922+0000 7fa9b5ffb640 1 -- 172.21.15.16:0/1996545269 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b8036070 con 0x55c0210d7b90 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.922+0000 7fa9b5ffb640 1 -- 172.21.15.16:0/1996545269 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80042a0 con 0x55c0210d7b90 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 -- 172.21.15.16:0/1996545269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 msgr2=0x55c0210d0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c0210d0b90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa9b802aee0 tx=0x7fa9b802a8f0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 -- 172.21.15.16:0/1996545269 shutdown_connections 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c0210d0b90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/1996545269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.923+0000 7fa9c8fad740 1 -- 172.21.15.16:0/1996545269 >> 172.21.15.16:0/1996545269 conn(0x55c0210cd6a0 msgr2=0x55c0210d3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.925+0000 7fa9c8fad740 1 -- 172.21.15.16:0/1996545269 shutdown_connections 2024-04-25T01:32:25.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.925+0000 7fa9c8fad740 1 -- 172.21.15.16:0/1996545269 wait complete. 2024-04-25T01:32:25.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.926+0000 7fa9c8fad740 1 -- start start 2024-04-25T01:32:25.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 Processor -- start 2024-04-25T01:32:25.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210e0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c02114caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01ad630 con 0x55c0210d2030 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01aeaa0 con 0x55c0210d7b90 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.927+0000 7fa9c8fad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01add00 con 0x55c0210d2a20 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9c8fad740 10 client.0 _ll_register_callbacks cb 0x7fa9a8097310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9bffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210e0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9b7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:55528/0 (socket says 172.21.15.16:55528) 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9b7fff640 1 -- 172.21.15.16:0/3610434937 learned_addr learned my addr 172.21.15.16:0/3610434937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9c483b640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c02114caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9bffff640 1 -- 172.21.15.16:0/3610434937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 msgr2=0x55c0210e0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9bffff640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9bffff640 1 -- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 msgr2=0x55c02114caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9bffff640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c02114caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9bffff640 1 -- 172.21.15.16:0/3610434937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c02114d090 con 0x55c0210d2030 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.928+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.929+0000 7fa9c483b640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c02114caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.929+0000 7fa9bffff640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210e0a70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000fd60 tx=0x7fa9b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82023c20b9a5c51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.929+0000 7fa99f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.929+0000 7fa99f7fe640 1 -- 172.21.15.16:0/3610434937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0016070 con 0x55c0210d2030 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.929+0000 7fa99f7fe640 1 -- 172.21.15.16:0/3610434937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa9b0028770 con 0x55c0210d2030 2024-04-25T01:32:25.908 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.929+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9b8036070 con 0x55c0210d2030 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.930+0000 7fa99f7fe640 1 -- 172.21.15.16:0/3610434937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa9b001c4a0 con 0x55c0210d2030 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.930+0000 7fa9c8fad740 10 client.15666 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.930+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c02114e410 con 0x55c0210d2030 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.930+0000 7fa99f7fe640 1 -- 172.21.15.16:0/3610434937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b8036070 con 0x55c0210d2030 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.930+0000 7fa99f7fe640 1 -- 172.21.15.16:0/3610434937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa9b003b730 con 0x55c0210d2030 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.931+0000 7fa9c8fad740 10 client.15666 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.931+0000 7fa9c8fad740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.932+0000 7fa9c8fad740 20 client.15666 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.932+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c02114edb0 0x55c0211511a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.932+0000 7fa9c8fad740 4 client.15666 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:32:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.932+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c02103d540 con 0x55c02114edb0 2024-04-25T01:32:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.932+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c02114edb0 0x55c0211511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.933+0000 7fa9b7fff640 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c02114edb0 0x55c0211511a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8021ff09ef57e438 server_cookie=67a3055060d92bac in_seq=0 out_seq=0 2024-04-25T01:32:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.933+0000 7fa99f7fe640 10 client.15666 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.938+0000 7fa99f7fe640 1 -- 172.21.15.16:0/3610434937 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c02103d540 con 0x55c02114edb0 2024-04-25T01:32:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.938+0000 7fa99f7fe640 10 client.15666 handle_command_reply: tid=0 2024-04-25T01:32:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.938+0000 7fa9c8fad740 1 client.15666 shutdown 2024-04-25T01:32:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.938+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 msgr2=0x55c0210e0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.938+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210e0a70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000fd60 tx=0x7fa9b00046c0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 shutdown_connections 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c02114edb0 0x55c0211511a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0210d7b90 0x55c02114caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa99f7fe640 0 client.15666 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0210d2a20 0x55c0210e0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 --2- 172.21.15.16:0/3610434937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0210d2030 0x55c0210e0a70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 >> 172.21.15.16:0/3610434937 conn(0x55c0210cd6a0 msgr2=0x55c0210cc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 shutdown_connections 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 1 -- 172.21.15.16:0/3610434937 wait complete. 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.939+0000 7fa9c8fad740 20 client.15666 trim_cache size 0 max 16384 2024-04-25T01:32:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:32:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:32:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:32:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa9ac0a2d80 msgr2=0x7fa9ac0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa9ac0a2d80 0x7fa9ac0a5240 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000f4d0 tx=0x7fa9b000a380 comp rx=0 tx=0).stop 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c0075780 msgr2=0x7fa9c019e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c0075780 0x7fa9c019e2b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800e9f0 tx=0x7fa9a800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 shutdown_connections 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa9ac0a2d80 0x7fa9ac0a5240 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9c010dba0 0x7fa9c01a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c0075780 0x7fa9c019e2b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 --2- 172.21.15.16:0/2788568848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9c00741a0 0x7fa9c019dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.940+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 >> 172.21.15.16:0/2788568848 conn(0x7fa9c00fcb40 msgr2=0x7fa9c0100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.941+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 shutdown_connections 2024-04-25T01:32:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:25.941+0000 7fa9c503c640 1 -- 172.21.15.16:0/2788568848 wait complete. 2024-04-25T01:32:27.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:26 smithi038 ceph-mon[32841]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:32:27.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:26 smithi156 ceph-mon[32631]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:32:27.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:26 smithi016 ceph-mon[29153]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:32:29.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:28 smithi038 ceph-mon[32841]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-04-25T01:32:29.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:28 smithi156 ceph-mon[32631]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-04-25T01:32:29.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:28 smithi016 ceph-mon[29153]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-04-25T01:32:31.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:30 smithi038 ceph-mon[32841]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:31.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:30 smithi156 ceph-mon[32631]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:31.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:30 smithi016 ceph-mon[29153]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:33.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:32 smithi038 ceph-mon[32841]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:32:33.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:32 smithi156 ceph-mon[32631]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:32:33.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:32 smithi016 ceph-mon[29153]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:32:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:34 smithi016 ceph-mon[29153]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T01:32:35.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:34 smithi038 ceph-mon[32841]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T01:32:35.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:34 smithi156 ceph-mon[32631]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T01:32:37.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:36 smithi038 ceph-mon[32841]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T01:32:37.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:36 smithi156 ceph-mon[32631]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T01:32:37.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:36 smithi016 ceph-mon[29153]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T01:32:39.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:38 smithi038 ceph-mon[32841]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:39.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:38 smithi156 ceph-mon[32631]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:39.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:38 smithi016 ceph-mon[29153]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:32:41.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:40 smithi038 ceph-mon[32841]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T01:32:41.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:40 smithi156 ceph-mon[32631]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T01:32:41.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:40 smithi016 ceph-mon[29153]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 -- 172.21.15.16:0/3626070585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19541006a0 msgr2=0x7f1954100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 --2- 172.21.15.16:0/3626070585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19541006a0 0x7f1954100ac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1944009900 tx=0x7f194402a400 comp rx=0 tx=0).stop 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 -- 172.21.15.16:0/3626070585 shutdown_connections 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 --2- 172.21.15.16:0/3626070585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1954102250 0x7f1954108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 --2- 172.21.15.16:0/3626070585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 0x7f1954101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 --2- 172.21.15.16:0/3626070585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19541006a0 0x7f1954100ac0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 -- 172.21.15.16:0/3626070585 >> 172.21.15.16:0/3626070585 conn(0x7f19540fbc10 msgr2=0x7f19540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 -- 172.21.15.16:0/3626070585 shutdown_connections 2024-04-25T01:32:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.229+0000 7f195a59d640 1 -- 172.21.15.16:0/3626070585 wait complete. 2024-04-25T01:32:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.230+0000 7f195a59d640 1 Processor -- start 2024-04-25T01:32:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.230+0000 7f195a59d640 1 -- start start 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.230+0000 7f195a59d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 0x7f1954199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f195a59d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 0x7f1954199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f195a59d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 0x7f19541a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f19537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 0x7f1954199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f19537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 0x7f1954199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:51244/0 (socket says 172.21.15.16:51244) 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f1958b3a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 0x7f19541a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f1958b3a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 0x7f19541a0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:47746/0 (socket says 172.21.15.16:47746) 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f1953fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 0x7f1954199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f1953fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 0x7f1954199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38924/0 (socket says 172.21.15.16:38924) 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f1958b3a640 1 -- 172.21.15.16:0/3226825402 learned_addr learned my addr 172.21.15.16:0/3226825402 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f195410d350 con 0x7f19541006a0 2024-04-25T01:32:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.231+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f195410d1d0 con 0x7f1954102250 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f195410d4d0 con 0x7f1954101890 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f1958b3a640 1 -- 172.21.15.16:0/3226825402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 msgr2=0x7f1954199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f1958b3a640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 0x7f1954199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f1958b3a640 1 -- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 msgr2=0x7f1954199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f1958b3a640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 0x7f1954199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f1958b3a640 1 -- 172.21.15.16:0/3226825402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19541a1590 con 0x7f1954102250 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.232+0000 7f1953fff640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 0x7f1954199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.233+0000 7f1958b3a640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 0x7f19541a0f20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f194c00eaa0 tx=0x7f194c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.233+0000 7f19517fa640 1 -- 172.21.15.16:0/3226825402 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c00cdb0 con 0x7f1954102250 2024-04-25T01:32:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.233+0000 7f19517fa640 1 -- 172.21.15.16:0/3226825402 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f194c005020 con 0x7f1954102250 2024-04-25T01:32:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.233+0000 7f19517fa640 1 -- 172.21.15.16:0/3226825402 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0116d0 con 0x7f1954102250 2024-04-25T01:32:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.233+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19541a1880 con 0x7f1954102250 2024-04-25T01:32:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.234+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1954102cf0 con 0x7f1954102250 2024-04-25T01:32:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.237+0000 7f19517fa640 1 -- 172.21.15.16:0/3226825402 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f194c0051d0 con 0x7f1954102250 2024-04-25T01:32:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.238+0000 7f19517fa640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f193c0a2da0 0x7f193c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.238+0000 7f19517fa640 1 -- 172.21.15.16:0/3226825402 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f194c0cf210 con 0x7f1954102250 2024-04-25T01:32:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.238+0000 7f195d382740 1 Processor -- start 2024-04-25T01:32:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.238+0000 7f1953fff640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f193c0a2da0 0x7f193c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 -- start start 2024-04-25T01:32:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4e840 0x55b0e3f53340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f55e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b0e3eb1cb0 con 0x55b0e3f4de50 2024-04-25T01:32:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b0e3e5a5b0 con 0x55b0e3f4e840 2024-04-25T01:32:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f195d382740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b0e3eadc70 con 0x55b0e3f539b0 2024-04-25T01:32:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f1953fff640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f193c0a2da0 0x7f193c0a5260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f194402a920 tx=0x7f1944000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f19537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4e840 0x55b0e3f53340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f1958b3a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f55e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f1958b3a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f55e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:51248/0 (socket says 172.21.15.16:51248) 2024-04-25T01:32:42.217 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f1958b3a640 1 -- 172.21.15.16:0/3726667404 learned_addr learned my addr 172.21.15.16:0/3726667404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.239+0000 7f1953fff640 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f1958b3a640 1 -- 172.21.15.16:0/3726667404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4e840 msgr2=0x55b0e3f53340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f1958b3a640 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4e840 0x55b0e3f53340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f1958b3a640 1 -- 172.21.15.16:0/3726667404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4de50 msgr2=0x55b0e3f4e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f1958b3a640 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f1958b3a640 1 -- 172.21.15.16:0/3726667404 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0e3bc6210 con 0x55b0e3f539b0 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f19537fe640 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4e840 0x55b0e3f53340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f1958b3a640 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f55e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f194c097eb0 tx=0x7f194c09fe00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e89e0a5e1d042f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f193a7fc640 1 -- 172.21.15.16:0/3726667404 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c01f040 con 0x55b0e3f539b0 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f193a7fc640 1 -- 172.21.15.16:0/3726667404 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f194c099d50 con 0x55b0e3f539b0 2024-04-25T01:32:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.240+0000 7f193a7fc640 1 -- 172.21.15.16:0/3726667404 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0979c0 con 0x55b0e3f539b0 2024-04-25T01:32:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 -- 172.21.15.16:0/3726667404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 msgr2=0x55b0e3f55e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f55e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f194c097eb0 tx=0x7f194c09fe00 comp rx=0 tx=0).stop 2024-04-25T01:32:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 -- 172.21.15.16:0/3726667404 shutdown_connections 2024-04-25T01:32:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f55e40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4e840 0x55b0e3f53340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 --2- 172.21.15.16:0/3726667404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.242+0000 7f195d382740 1 -- 172.21.15.16:0/3726667404 >> 172.21.15.16:0/3726667404 conn(0x55b0e3f494c0 msgr2=0x55b0e3f498c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.243+0000 7f195d382740 1 -- 172.21.15.16:0/3726667404 shutdown_connections 2024-04-25T01:32:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.243+0000 7f195d382740 1 -- 172.21.15.16:0/3726667404 wait complete. 2024-04-25T01:32:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.243+0000 7f195d382740 1 -- start start 2024-04-25T01:32:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 Processor -- start 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f69020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b0e3f67910 con 0x55b0e3f4e840 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b0e3e6db50 con 0x55b0e3f4de50 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b0e3f67a90 con 0x55b0e3f539b0 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f195d382740 10 client.0 _ll_register_callbacks cb 0x55b0e3ea8ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f1953fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f69020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f1953fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f69020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:47762/0 (socket says 172.21.15.16:47762) 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.244+0000 7f1953fff640 1 -- 172.21.15.16:0/2676477801 learned_addr learned my addr 172.21.15.16:0/2676477801 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f19537fe640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f1958b3a640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f1958b3a640 1 -- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 msgr2=0x55b0e3f69020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f1958b3a640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f69020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f1958b3a640 1 -- 172.21.15.16:0/2676477801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 msgr2=0x55b0e3f69560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f1958b3a640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f69560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.245+0000 7f1958b3a640 1 -- 172.21.15.16:0/2676477801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0e3c92620 con 0x55b0e3f539b0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.246+0000 7f1958b3a640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f705a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f194c097eb0 tx=0x7f194c097a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e5e87b026c2eb53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.246+0000 7f19337fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.246+0000 7f19337fe640 1 -- 172.21.15.16:0/2676477801 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0b8070 con 0x55b0e3f539b0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.246+0000 7f19337fe640 1 -- 172.21.15.16:0/2676477801 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f194c09ab20 con 0x55b0e3f539b0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.246+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f194c099d50 con 0x55b0e3f539b0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f19337fe640 1 -- 172.21.15.16:0/2676477801 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f194c01f040 con 0x55b0e3f539b0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f195d382740 10 client.25318 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b0e3f67980 con 0x55b0e3f539b0 2024-04-25T01:32:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f19337fe640 1 -- 172.21.15.16:0/2676477801 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f194c099d50 con 0x55b0e3f539b0 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f19337fe640 1 -- 172.21.15.16:0/2676477801 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f194c00f040 con 0x55b0e3f539b0 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f195d382740 10 client.25318 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f195d382740 10 client.25318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.247+0000 7f195d382740 20 client.25318 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.248+0000 7f195d382740 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3f74ba0 0x55b0e3f76f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.248+0000 7f195d382740 4 client.25318 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.248+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0e3ebf1f0 con 0x55b0e3f74ba0 2024-04-25T01:32:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.248+0000 7f1953fff640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3f74ba0 0x55b0e3f76f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.248+0000 7f19337fe640 10 client.25318 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.248+0000 7f1953fff640 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3f74ba0 0x55b0e3f76f90 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2d451a3572a0efb server_cookie=ed0b6be9dd89b4ed in_seq=0 out_seq=0 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.249+0000 7f19337fe640 1 -- 172.21.15.16:0/2676477801 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0e3ebf1f0 con 0x55b0e3f74ba0 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.249+0000 7f19337fe640 10 client.25318 handle_command_reply: tid=0 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.249+0000 7f195d382740 1 client.25318 shutdown 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.249+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 msgr2=0x55b0e3f705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.249+0000 7f195d382740 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f705a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f194c097eb0 tx=0x7f194c097a50 comp rx=0 tx=0).stop 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 shutdown_connections 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f539b0 0x55b0e3f705a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f69560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3f74ba0 0x55b0e3f76f90 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 --2- 172.21.15.16:0/2676477801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f69020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f19337fe640 0 client.25318 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 >> 172.21.15.16:0/2676477801 conn(0x55b0e3f494c0 msgr2=0x55b0e3f485c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 shutdown_connections 2024-04-25T01:32:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 1 -- 172.21.15.16:0/2676477801 wait complete. 2024-04-25T01:32:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.250+0000 7f195d382740 20 client.25318 trim_cache size 0 max 16384 2024-04-25T01:32:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.280+0000 7f195d382740 1 Processor -- start 2024-04-25T01:32:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.280+0000 7f195d382740 1 -- start start 2024-04-25T01:32:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.280+0000 7f195d382740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.280+0000 7f195d382740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4e840 0x55b0e3f70560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.281+0000 7f1953fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.281+0000 7f195d382740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f539b0 0x55b0e3f50000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.281+0000 7f195d382740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19541a9180 con 0x55b0e3f539b0 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.281+0000 7f195d382740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19541a6a50 con 0x55b0e3f4de50 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.281+0000 7f195d382740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19541a48c0 con 0x55b0e3f4e840 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f19537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4e840 0x55b0e3f70560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f19537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4e840 0x55b0e3f70560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:51270/0 (socket says 172.21.15.16:51270) 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f19537fe640 1 -- 172.21.15.16:0/2238234258 learned_addr learned my addr 172.21.15.16:0/2238234258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1958b3a640 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f539b0 0x55b0e3f50000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1953fff640 1 -- 172.21.15.16:0/2238234258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4e840 msgr2=0x55b0e3f70560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1953fff640 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4e840 0x55b0e3f70560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1953fff640 1 -- 172.21.15.16:0/2238234258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f539b0 msgr2=0x55b0e3f50000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1953fff640 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f539b0 0x55b0e3f50000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1953fff640 1 -- 172.21.15.16:0/2238234258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0e3f77570 con 0x55b0e3f4de50 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1958b3a640 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f539b0 0x55b0e3f50000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.282+0000 7f1953fff640 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4fac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55b0e3f777d0 tx=0x7f194403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e8b28e10a27a3db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.283+0000 7f1939ffb640 1 -- 172.21.15.16:0/2238234258 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194405c070 con 0x55b0e3f4de50 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.283+0000 7f1939ffb640 1 -- 172.21.15.16:0/2238234258 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1944003aa0 con 0x55b0e3f4de50 2024-04-25T01:32:42.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.283+0000 7f1939ffb640 1 -- 172.21.15.16:0/2238234258 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944035040 con 0x55b0e3f4de50 2024-04-25T01:32:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 -- 172.21.15.16:0/2238234258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 msgr2=0x55b0e3f4fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4fac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55b0e3f777d0 tx=0x7f194403d000 comp rx=0 tx=0).stop 2024-04-25T01:32:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 -- 172.21.15.16:0/2238234258 shutdown_connections 2024-04-25T01:32:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f539b0 0x55b0e3f50000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4e840 0x55b0e3f70560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 --2- 172.21.15.16:0/2238234258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f4de50 0x55b0e3f4fac0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 -- 172.21.15.16:0/2238234258 >> 172.21.15.16:0/2238234258 conn(0x55b0e3f494c0 msgr2=0x55b0e3f69020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.284+0000 7f195d382740 1 -- 172.21.15.16:0/2238234258 shutdown_connections 2024-04-25T01:32:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.285+0000 7f195d382740 1 -- 172.21.15.16:0/2238234258 wait complete. 2024-04-25T01:32:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.285+0000 7f195d382740 1 -- start start 2024-04-25T01:32:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.286+0000 7f195d382740 1 Processor -- start 2024-04-25T01:32:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.286+0000 7f195d382740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 0x55b0e3f6d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f195d382740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f6d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f195d382740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 0x55b0e3fd9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f195d382740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f19541a4f10 con 0x55b0e3f4e840 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f195d382740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19541a6380 con 0x55b0e3f539b0 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f195d382740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19541a55e0 con 0x55b0e3f4de50 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f195d382740 10 client.0 _ll_register_callbacks cb 0x7f194c09c220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f1953fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 0x55b0e3f6d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f19537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f6d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f1958b3a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 0x55b0e3fd9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f1953fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 0x55b0e3f6d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:51278/0 (socket says 172.21.15.16:51278) 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f19537fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f6d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38976/0 (socket says 172.21.15.16:38976) 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f1953fff640 1 -- 172.21.15.16:0/853056012 learned_addr learned my addr 172.21.15.16:0/853056012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.287+0000 7f1958b3a640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 0x55b0e3fd9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:47770/0 (socket says 172.21.15.16:47770) 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1958b3a640 1 -- 172.21.15.16:0/853056012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 msgr2=0x55b0e3f6d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1958b3a640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 0x55b0e3f6d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1958b3a640 1 -- 172.21.15.16:0/853056012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 msgr2=0x55b0e3f6d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1958b3a640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f6d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1958b3a640 1 -- 172.21.15.16:0/853056012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0e3f6ded0 con 0x55b0e3f539b0 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1953fff640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 0x55b0e3f6d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:42.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f1958b3a640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 0x55b0e3fd9090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f194c09b9e0 tx=0x7f194c099f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a54b3825457be06f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f19337fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f19337fe640 1 -- 172.21.15.16:0/853056012 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0b8070 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1944003aa0 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f19337fe640 1 -- 172.21.15.16:0/853056012 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f194c0980c0 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f19337fe640 1 -- 172.21.15.16:0/853056012 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f194c07f050 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f195d382740 10 client.25361 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.288+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b0e3f6ded0 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f19337fe640 1 -- 172.21.15.16:0/853056012 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1944003aa0 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f19337fe640 1 -- 172.21.15.16:0/853056012 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f194c004c50 con 0x55b0e3f539b0 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f195d382740 10 client.25361 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f195d382740 10 client.25361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f195d382740 20 client.25361 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f195d382740 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3fdad60 0x55b0e3fdd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:42.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f195d382740 4 client.25361 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:32:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0e3eb9540 con 0x55b0e3fdad60 2024-04-25T01:32:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.289+0000 7f1953fff640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3fdad60 0x55b0e3fdd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.290+0000 7f19337fe640 10 client.25361 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.290+0000 7f1953fff640 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3fdad60 0x55b0e3fdd180 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0fcd101680ba634 server_cookie=46849f63bfe74c31 in_seq=0 out_seq=0 2024-04-25T01:32:42.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.302+0000 7f19337fe640 1 -- 172.21.15.16:0/853056012 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0e3eb9540 con 0x55b0e3fdad60 2024-04-25T01:32:42.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.302+0000 7f19337fe640 10 client.25361 handle_command_reply: tid=0 2024-04-25T01:32:42.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.302+0000 7f195d382740 1 client.25361 shutdown 2024-04-25T01:32:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 msgr2=0x55b0e3fd9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 0x55b0e3fd9090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f194c09b9e0 tx=0x7f194c099f60 comp rx=0 tx=0).stop 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 shutdown_connections 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b0e3fdad60 0x55b0e3fdd180 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0e3f539b0 0x55b0e3fd9090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f19337fe640 0 client.25361 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b0e3f4e840 0x55b0e3f6d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 --2- 172.21.15.16:0/853056012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0e3f4de50 0x55b0e3f6d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 >> 172.21.15.16:0/853056012 conn(0x55b0e3f494c0 msgr2=0x55b0e3f47d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 shutdown_connections 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.303+0000 7f195d382740 1 -- 172.21.15.16:0/853056012 wait complete. 2024-04-25T01:32:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.304+0000 7f195d382740 20 client.25361 trim_cache size 0 max 16384 2024-04-25T01:32:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:32:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:32:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:32:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:32:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.305+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f193c0a2da0 msgr2=0x7f193c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.305+0000 7f195a59d640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f193c0a2da0 0x7f193c0a5260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f194402a920 tx=0x7f1944000bc0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.305+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 msgr2=0x7f19541a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.305+0000 7f195a59d640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 0x7f19541a0f20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f194c00eaa0 tx=0x7f194c00ef60 comp rx=0 tx=0).stop 2024-04-25T01:32:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.305+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 shutdown_connections 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.305+0000 7f195a59d640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f193c0a2da0 0x7f193c0a5260 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.306+0000 7f195a59d640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102250 0x7f19541a0f20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.306+0000 7f195a59d640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1954101890 0x7f1954199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.306+0000 7f195a59d640 1 --2- 172.21.15.16:0/3226825402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19541006a0 0x7f1954199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.306+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 >> 172.21.15.16:0/3226825402 conn(0x7f19540fbc10 msgr2=0x7f19541094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.306+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 shutdown_connections 2024-04-25T01:32:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:42.306+0000 7f195a59d640 1 -- 172.21.15.16:0/3226825402 wait complete. 2024-04-25T01:32:43.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:42 smithi038 ceph-mon[32841]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:32:43.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:42 smithi156 ceph-mon[32631]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:32:43.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:42 smithi016 ceph-mon[29153]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:32:45.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:44 smithi038 ceph-mon[32841]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T01:32:45.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:44 smithi156 ceph-mon[32631]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T01:32:45.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:44 smithi016 ceph-mon[29153]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T01:32:47.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:46 smithi038 ceph-mon[32841]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:32:47.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:46 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:32:47.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:46 smithi156 ceph-mon[32631]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:32:47.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:46 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:32:47.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:46 smithi016 ceph-mon[29153]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:32:47.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:46 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:32:49.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:48 smithi038 ceph-mon[32841]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:49.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:49.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:49.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:32:49.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:48 smithi156 ceph-mon[32631]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:49.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:49.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:49.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:32:49.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:48 smithi016 ceph-mon[29153]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:32:49.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:49.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:49.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:32:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:50 smithi038 ceph-mon[32841]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:32:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:50 smithi156 ceph-mon[32631]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:32:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:50 smithi016 ceph-mon[29153]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:32:53.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:52 smithi038 ceph-mon[32841]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:32:53.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:52 smithi156 ceph-mon[32631]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:32:53.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:52 smithi016 ceph-mon[29153]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:32:55.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:54 smithi038 ceph-mon[32841]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-04-25T01:32:55.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:54 smithi156 ceph-mon[32631]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-04-25T01:32:55.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:54 smithi016 ceph-mon[29153]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.478+0000 7f288e0b2640 1 -- 172.21.15.16:0/2969819669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2888107520 msgr2=0x7f2888109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.478+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2969819669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2888107520 0x7f2888109910 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2880009ca0 tx=0x7f288002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 -- 172.21.15.16:0/2969819669 shutdown_connections 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2969819669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2888107520 0x7f2888109910 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2969819669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2888104bf0 0x7f2888106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2969819669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28881022c0 0x7f28881046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 -- 172.21.15.16:0/2969819669 >> 172.21.15.16:0/2969819669 conn(0x7f28880fbc10 msgr2=0x7f28880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 -- 172.21.15.16:0/2969819669 shutdown_connections 2024-04-25T01:32:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.479+0000 7f288e0b2640 1 -- 172.21.15.16:0/2969819669 wait complete. 2024-04-25T01:32:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.480+0000 7f288e0b2640 1 Processor -- start 2024-04-25T01:32:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.480+0000 7f288e0b2640 1 -- start start 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.480+0000 7f288e0b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 0x7f2888101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f288e0b2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2888104bf0 0x7f2888100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f288e0b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2888107520 0x7f28881007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f288e0b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f288810d320 con 0x7f28881022c0 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f28877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 0x7f2888101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f28877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 0x7f2888101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34510/0 (socket says 172.21.15.16:34510) 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f28877fe640 1 -- 172.21.15.16:0/2887681083 learned_addr learned my addr 172.21.15.16:0/2887681083 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f2886ffd640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2888104bf0 0x7f2888100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f288810d1a0 con 0x7f2888104bf0 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f288810d4a0 con 0x7f2888107520 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f2887fff640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2888107520 0x7f28881007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f28877fe640 1 -- 172.21.15.16:0/2887681083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2888107520 msgr2=0x7f28881007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f28877fe640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2888107520 0x7f28881007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.481+0000 7f28877fe640 1 -- 172.21.15.16:0/2887681083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2888104bf0 msgr2=0x7f2888100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.482+0000 7f28877fe640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2888104bf0 0x7f2888100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.482+0000 7f28877fe640 1 -- 172.21.15.16:0/2887681083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2888100cf0 con 0x7f28881022c0 2024-04-25T01:32:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.482+0000 7f28877fe640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 0x7f2888101c30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f287800e9f0 tx=0x7f287800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.482+0000 7f2884ff9640 1 -- 172.21.15.16:0/2887681083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287800cd80 con 0x7f28881022c0 2024-04-25T01:32:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.483+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2888100fe0 con 0x7f28881022c0 2024-04-25T01:32:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.483+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28881a63f0 con 0x7f28881022c0 2024-04-25T01:32:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.483+0000 7f2884ff9640 1 -- 172.21.15.16:0/2887681083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28780108e0 con 0x7f28881022c0 2024-04-25T01:32:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.483+0000 7f2884ff9640 1 -- 172.21.15.16:0/2887681083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2878026060 con 0x7f28881022c0 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.487+0000 7f2884ff9640 1 -- 172.21.15.16:0/2887681083 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f2878018850 con 0x7f28881022c0 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 Processor -- start 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 -- start start 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f0840 0x558fd19f5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd19f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558fd1953cb0 con 0x558fd19efe50 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558fd18fc5b0 con 0x558fd19f59b0 2024-04-25T01:32:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2890dbd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558fd194fc70 con 0x558fd19f0840 2024-04-25T01:32:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2887fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2886ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd19f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f28877fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f0840 0x558fd19f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2886ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd19f7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:44486/0 (socket says 172.21.15.16:44486) 2024-04-25T01:32:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.488+0000 7f2886ffd640 1 -- 172.21.15.16:0/96008507 learned_addr learned my addr 172.21.15.16:0/96008507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.489+0000 7f2884ff9640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f28700a2d30 0x7f28700a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2887fff640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f28700a2d30 0x7f28700a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2884ff9640 1 -- 172.21.15.16:0/2887681083 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f28780cf3f0 con 0x7f28881022c0 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2887fff640 1 -- 172.21.15.16:0/96008507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f0840 msgr2=0x558fd19f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2887fff640 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f0840 0x558fd19f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2887fff640 1 -- 172.21.15.16:0/96008507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 msgr2=0x558fd19f7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2887fff640 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd19f7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.490+0000 7f2887fff640 1 -- 172.21.15.16:0/96008507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fd1668210 con 0x558fd19efe50 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.491+0000 7f2886ffd640 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd19f7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:55.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.491+0000 7f2887fff640 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f0270 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f288002ade0 tx=0x7f2880031450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e75477a119f44c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.491+0000 7f286dffb640 1 -- 172.21.15.16:0/96008507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880031b40 con 0x558fd19efe50 2024-04-25T01:32:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.491+0000 7f286dffb640 1 -- 172.21.15.16:0/96008507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2880036070 con 0x558fd19efe50 2024-04-25T01:32:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.491+0000 7f286dffb640 1 -- 172.21.15.16:0/96008507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880004660 con 0x558fd19efe50 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2887fff640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f28700a2d30 0x7f28700a51f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f28800049d0 tx=0x7f2880004800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 -- 172.21.15.16:0/96008507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 msgr2=0x558fd19f0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f0270 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f288002ade0 tx=0x7f2880031450 comp rx=0 tx=0).stop 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 -- 172.21.15.16:0/96008507 shutdown_connections 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd19f7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f0840 0x558fd19f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 --2- 172.21.15.16:0/96008507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f0270 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.492+0000 7f2890dbd740 1 -- 172.21.15.16:0/96008507 >> 172.21.15.16:0/96008507 conn(0x558fd19eb4c0 msgr2=0x558fd19eb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.493+0000 7f2890dbd740 1 -- 172.21.15.16:0/96008507 shutdown_connections 2024-04-25T01:32:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.493+0000 7f2890dbd740 1 -- 172.21.15.16:0/96008507 wait complete. 2024-04-25T01:32:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.493+0000 7f2890dbd740 1 -- start start 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.493+0000 7f2890dbd740 1 Processor -- start 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.494+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.494+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd19fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.494+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f59b0 0x558fd1a015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.494+0000 7f2886ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd19fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.494+0000 7f2886ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd19fa5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:44492/0 (socket says 172.21.15.16:44492) 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.495+0000 7f2886ffd640 1 -- 172.21.15.16:0/187201507 learned_addr learned my addr 172.21.15.16:0/187201507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.495+0000 7f2887fff640 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f59b0 0x558fd1a015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.495+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558fd19f87e0 con 0x558fd19f59b0 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.495+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558fd190fb50 con 0x558fd19f0840 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.495+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558fd19f8960 con 0x558fd19efe50 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2886ffd640 1 -- 172.21.15.16:0/187201507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 msgr2=0x558fd19fa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2886ffd640 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2886ffd640 1 -- 172.21.15.16:0/187201507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f59b0 msgr2=0x558fd1a015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2886ffd640 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f59b0 0x558fd1a015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2886ffd640 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fd1734620 con 0x558fd19f0840 2024-04-25T01:32:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2886ffd640 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd19fa5a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f287c012760 tx=0x7f287c012c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c794411c8e34210 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:32:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.497+0000 7f2866ffd640 1 -- 172.21.15.16:0/187201507 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c00f920 con 0x558fd19f0840 2024-04-25T01:32:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.497+0000 7f2866ffd640 1 -- 172.21.15.16:0/187201507 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f287c00fac0 con 0x558fd19f0840 2024-04-25T01:32:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.496+0000 7f2890dbd740 10 client.0 _ll_register_callbacks cb 0x558fd194aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.497+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2880036070 con 0x558fd19f0840 2024-04-25T01:32:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.498+0000 7f2866ffd640 1 -- 172.21.15.16:0/187201507 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f287c00fcc0 con 0x558fd19f0840 2024-04-25T01:32:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.498+0000 7f2890dbd740 10 client.25373 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.498+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=22}) -- 0x558fd1a04850 con 0x558fd19f0840 2024-04-25T01:32:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2866ffd640 1 -- 172.21.15.16:0/187201507 <== mon.2 v2:172.21.15.38:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f287c026470 con 0x558fd19f0840 2024-04-25T01:32:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2890dbd740 10 client.25373 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2890dbd740 10 client.25373 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2890dbd740 20 client.25373 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2890dbd740 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a05e20 0x558fd1a082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2890dbd740 4 client.25373 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.499+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2880036070 con 0x558fd1a05e20 2024-04-25T01:32:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.500+0000 7f28877fe640 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a05e20 0x558fd1a082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.500+0000 7f28877fe640 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a05e20 0x558fd1a082e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4134e848ec5d96a server_cookie=585a7768d542a094 in_seq=0 out_seq=0 2024-04-25T01:32:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.500+0000 7f2866ffd640 10 client.25373 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.501+0000 7f2866ffd640 1 -- 172.21.15.16:0/187201507 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f2880036070 con 0x558fd1a05e20 2024-04-25T01:32:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.501+0000 7f2866ffd640 10 client.25373 handle_command_reply: tid=0 2024-04-25T01:32:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.501+0000 7f2890dbd740 1 client.25373 shutdown 2024-04-25T01:32:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 msgr2=0x558fd19fa5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd19fa5a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f287c012760 tx=0x7f287c012c20 comp rx=0 tx=0).stop 2024-04-25T01:32:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 shutdown_connections 2024-04-25T01:32:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f59b0 0x558fd1a015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a05e20 0x558fd1a082e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2866ffd640 0 client.25373 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd19fa5a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 --2- 172.21.15.16:0/187201507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 >> 172.21.15.16:0/187201507 conn(0x558fd19eb4c0 msgr2=0x558fd19ea5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.502+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 shutdown_connections 2024-04-25T01:32:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.503+0000 7f2890dbd740 1 -- 172.21.15.16:0/187201507 wait complete. 2024-04-25T01:32:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.503+0000 7f2890dbd740 20 client.25373 trim_cache size 0 max 16384 2024-04-25T01:32:55.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.531+0000 7f2890dbd740 1 Processor -- start 2024-04-25T01:32:55.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.532+0000 7f2890dbd740 1 -- start start 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.532+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.532+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd1a015a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.532+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f59b0 0x558fd19f2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.532+0000 7f2890dbd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28881b1a20 con 0x558fd19efe50 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2890dbd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28881af2f0 con 0x558fd19f0840 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2886ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd1a015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f28877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2886ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd1a015a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:44508/0 (socket says 172.21.15.16:44508) 2024-04-25T01:32:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2886ffd640 1 -- 172.21.15.16:0/3175052085 learned_addr learned my addr 172.21.15.16:0/3175052085 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f28877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34536/0 (socket says 172.21.15.16:34536) 2024-04-25T01:32:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2887fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f59b0 0x558fd19f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2890dbd740 1 -- 172.21.15.16:0/3175052085 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28881ad140 con 0x558fd19f59b0 2024-04-25T01:32:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2887fff640 1 -- 172.21.15.16:0/3175052085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 msgr2=0x558fd1a015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2887fff640 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd1a015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2887fff640 1 -- 172.21.15.16:0/3175052085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 msgr2=0x558fd19f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2887fff640 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.533+0000 7f2887fff640 1 -- 172.21.15.16:0/3175052085 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fd19f0c20 con 0x558fd19f59b0 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.534+0000 7f2887fff640 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f59b0 0x558fd19f2100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f288002ade0 tx=0x7f2880052000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5471db48ac55eef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.534+0000 7f286d7fa640 1 -- 172.21.15.16:0/3175052085 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880009800 con 0x558fd19f59b0 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.534+0000 7f286d7fa640 1 -- 172.21.15.16:0/3175052085 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2880061070 con 0x558fd19f59b0 2024-04-25T01:32:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.534+0000 7f286d7fa640 1 -- 172.21.15.16:0/3175052085 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880003970 con 0x558fd19f59b0 2024-04-25T01:32:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.535+0000 7f2890dbd740 1 -- 172.21.15.16:0/3175052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f59b0 msgr2=0x558fd19f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.535+0000 7f2890dbd740 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f59b0 0x558fd19f2100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f288002ade0 tx=0x7f2880052000 comp rx=0 tx=0).stop 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.537+0000 7f2890dbd740 1 -- 172.21.15.16:0/3175052085 shutdown_connections 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.537+0000 7f2890dbd740 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19f59b0 0x558fd19f2100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.537+0000 7f2890dbd740 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f0840 0x558fd1a015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.537+0000 7f2890dbd740 1 --2- 172.21.15.16:0/3175052085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19efe50 0x558fd19f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.537+0000 7f2890dbd740 1 -- 172.21.15.16:0/3175052085 >> 172.21.15.16:0/3175052085 conn(0x558fd19eb4c0 msgr2=0x558fd19eb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.538+0000 7f2890dbd740 1 -- 172.21.15.16:0/3175052085 shutdown_connections 2024-04-25T01:32:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.538+0000 7f2890dbd740 1 -- 172.21.15.16:0/3175052085 wait complete. 2024-04-25T01:32:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.538+0000 7f2890dbd740 1 -- start start 2024-04-25T01:32:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.539+0000 7f2890dbd740 1 Processor -- start 2024-04-25T01:32:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.539+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.540+0000 7f2890dbd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f0840 0x558fd19ff240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.540+0000 7f28877fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.540+0000 7f28877fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:49636/0 (socket says 172.21.15.16:49636) 2024-04-25T01:32:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.540+0000 7f28877fe640 1 -- 172.21.15.16:0/2874100194 learned_addr learned my addr 172.21.15.16:0/2874100194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:32:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.540+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd1a00250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.541+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28881ad7b0 con 0x558fd19f0840 2024-04-25T01:32:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.541+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28881aec20 con 0x558fd19f59b0 2024-04-25T01:32:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.542+0000 7f2886ffd640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f0840 0x558fd19ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.542+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28881ade80 con 0x558fd19efe50 2024-04-25T01:32:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.542+0000 7f2887fff640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd1a00250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.542+0000 7f2890dbd740 10 client.0 _ll_register_callbacks cb 0x7f287c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.543+0000 7f28877fe640 1 -- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 msgr2=0x558fd1a00250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.543+0000 7f28877fe640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd1a00250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.543+0000 7f28877fe640 1 -- 172.21.15.16:0/2874100194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f0840 msgr2=0x558fd19ff240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.543+0000 7f28877fe640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f0840 0x558fd19ff240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.543+0000 7f28877fe640 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fd1a00840 con 0x558fd19efe50 2024-04-25T01:32:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.543+0000 7f2887fff640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd1a00250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.544+0000 7f28877fe640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fed00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f287800cfd0 tx=0x7f2878009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df2eca47f50cc63b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.544+0000 7f2866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:32:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.547+0000 7f2866ffd640 1 -- 172.21.15.16:0/2874100194 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28780b9060 con 0x558fd19efe50 2024-04-25T01:32:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.547+0000 7f2866ffd640 1 -- 172.21.15.16:0/2874100194 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2878010a90 con 0x558fd19efe50 2024-04-25T01:32:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.547+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2880061070 con 0x558fd19efe50 2024-04-25T01:32:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.547+0000 7f2866ffd640 1 -- 172.21.15.16:0/2874100194 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f28780b2030 con 0x558fd19efe50 2024-04-25T01:32:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.549+0000 7f2890dbd740 10 client.25339 fetch_fsmap learned FSMap version 22 2024-04-25T01:32:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.549+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558fd1a6c4c0 con 0x558fd19efe50 2024-04-25T01:32:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2866ffd640 1 -- 172.21.15.16:0/2874100194 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28780b4070 con 0x558fd19efe50 2024-04-25T01:32:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2866ffd640 1 -- 172.21.15.16:0/2874100194 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f287809b0b0 con 0x558fd19efe50 2024-04-25T01:32:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2890dbd740 10 client.25339 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:32:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2890dbd740 10 client.25339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:32:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2890dbd740 20 client.25339 populate_metadata read hostname 'smithi016' 2024-04-25T01:32:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a6cdc0 0x558fd1a6f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2890dbd740 4 client.25339 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:32:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.551+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f2880061070 con 0x558fd1a6cdc0 2024-04-25T01:32:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.552+0000 7f2886ffd640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a6cdc0 0x558fd1a6f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.554+0000 7f2886ffd640 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a6cdc0 0x558fd1a6f1b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df2348c78e74d730 server_cookie=7fdb82104903d5aa in_seq=0 out_seq=0 2024-04-25T01:32:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.554+0000 7f2866ffd640 10 client.25339 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.563+0000 7f2866ffd640 1 -- 172.21.15.16:0/2874100194 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f2880061070 con 0x558fd1a6cdc0 2024-04-25T01:32:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.564+0000 7f2866ffd640 10 client.25339 handle_command_reply: tid=0 2024-04-25T01:32:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.564+0000 7f2890dbd740 1 client.25339 shutdown 2024-04-25T01:32:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.564+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 msgr2=0x558fd19fed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.564+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fed00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f287800cfd0 tx=0x7f2878009700 comp rx=0 tx=0).stop 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.565+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 shutdown_connections 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.565+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558fd1a6cdc0 0x558fd1a6f1b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd19f59b0 0x558fd1a00250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558fd19f0840 0x558fd19ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 1 --2- 172.21.15.16:0/2874100194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558fd19efe50 0x558fd19fed00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 >> 172.21.15.16:0/2874100194 conn(0x558fd19eb4c0 msgr2=0x558fd19fa020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2866ffd640 0 client.25339 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 shutdown_connections 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 1 -- 172.21.15.16:0/2874100194 wait complete. 2024-04-25T01:32:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.566+0000 7f2890dbd740 20 client.25339 trim_cache size 0 max 16384 2024-04-25T01:32:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:32:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:32:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:32:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:32:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f28700a2d30 msgr2=0x7f28700a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f28700a2d30 0x7f28700a51f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f28800049d0 tx=0x7f2880004800 comp rx=0 tx=0).stop 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 msgr2=0x7f2888101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 0x7f2888101c30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f287800e9f0 tx=0x7f287800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 shutdown_connections 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f28700a2d30 0x7f28700a51f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2888107520 0x7f28881007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2888104bf0 0x7f2888100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 --2- 172.21.15.16:0/2887681083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28881022c0 0x7f2888101c30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.569+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 >> 172.21.15.16:0/2887681083 conn(0x7f28880fbc10 msgr2=0x7f2888102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.570+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 shutdown_connections 2024-04-25T01:32:55.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:32:55.570+0000 7f288e0b2640 1 -- 172.21.15.16:0/2887681083 wait complete. 2024-04-25T01:32:57.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:56 smithi038 ceph-mon[32841]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 61 MiB/s wr, 988 op/s 2024-04-25T01:32:57.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:56 smithi156 ceph-mon[32631]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 61 MiB/s wr, 988 op/s 2024-04-25T01:32:57.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:56 smithi016 ceph-mon[29153]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 61 MiB/s wr, 988 op/s 2024-04-25T01:32:59.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:32:58 smithi038 ceph-mon[32841]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:32:59.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:32:58 smithi156 ceph-mon[32631]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:32:59.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:32:58 smithi016 ceph-mon[29153]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:33:01.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:00 smithi038 ceph-mon[32841]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T01:33:01.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:00 smithi156 ceph-mon[32631]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T01:33:01.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:00 smithi016 ceph-mon[29153]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T01:33:03.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:02 smithi038 ceph-mon[32841]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:03.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:02 smithi156 ceph-mon[32631]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:03.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:02 smithi016 ceph-mon[29153]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:05.748 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:05 smithi156 ceph-mon[32631]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:33:05.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:05 smithi016 ceph-mon[29153]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:33:05.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:05 smithi038 ceph-mon[32841]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T01:33:07.748 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:07 smithi156 ceph-mon[32631]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:33:07.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:07 smithi016 ceph-mon[29153]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:33:07.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:07 smithi038 ceph-mon[32841]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T01:33:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.724+0000 7f6f8cc38640 1 -- 172.21.15.16:0/2596620719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 msgr2=0x7f6f8810be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.724+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/2596620719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f8810be10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009be0 tx=0x7f6f8002a6c0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 -- 172.21.15.16:0/2596620719 shutdown_connections 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/2596620719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f8810be10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/2596620719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f88104bc0 0x7f6f88105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/2596620719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f881039d0 0x7f6f88103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 -- 172.21.15.16:0/2596620719 >> 172.21.15.16:0/2596620719 conn(0x7f6f880fef60 msgr2=0x7f6f88101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 -- 172.21.15.16:0/2596620719 shutdown_connections 2024-04-25T01:33:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.725+0000 7f6f8cc38640 1 -- 172.21.15.16:0/2596620719 wait complete. 2024-04-25T01:33:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.726+0000 7f6f8cc38640 1 Processor -- start 2024-04-25T01:33:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.726+0000 7f6f8cc38640 1 -- start start 2024-04-25T01:33:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8cc38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f881039d0 0x7f6f8819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8cc38640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f88104bc0 0x7f6f8819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8cc38640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f881a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8cc38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8810f4e0 con 0x7f6f881039d0 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8cc38640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8810f360 con 0x7f6f88104bc0 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8cc38640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8810f660 con 0x7f6f88105580 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f88104bc0 0x7f6f8819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f881a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f88104bc0 0x7f6f8819e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60528/0 (socket says 172.21.15.16:60528) 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f85d9b640 1 -- 172.21.15.16:0/949704501 learned_addr learned my addr 172.21.15.16:0/949704501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f881a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:47740/0 (socket says 172.21.15.16:47740) 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f8659c640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f881039d0 0x7f6f8819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 -- 172.21.15.16:0/949704501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f88104bc0 msgr2=0x7f6f8819e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f88104bc0 0x7f6f8819e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 -- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f881039d0 msgr2=0x7f6f8819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f881039d0 0x7f6f8819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.727+0000 7f6f86d9d640 1 -- 172.21.15.16:0/949704501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8806b290 con 0x7f6f88105580 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.728+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f881a5410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f80034040 tx=0x7f6f80030fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.728+0000 7f6f777fe640 1 -- 172.21.15.16:0/949704501 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80051070 con 0x7f6f88105580 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.728+0000 7f6f777fe640 1 -- 172.21.15.16:0/949704501 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f80037040 con 0x7f6f88105580 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.728+0000 7f6f777fe640 1 -- 172.21.15.16:0/949704501 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80036070 con 0x7f6f88105580 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.728+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8806b520 con 0x7f6f88105580 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.728+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8806b990 con 0x7f6f88105580 2024-04-25T01:33:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.731+0000 7f6f777fe640 1 -- 172.21.15.16:0/949704501 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6f80054070 con 0x7f6f88105580 2024-04-25T01:33:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.732+0000 7f6f8fa28740 1 Processor -- start 2024-04-25T01:33:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.732+0000 7f6f777fe640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f700a2c30 0x7f6f700a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f777fe640 1 -- 172.21.15.16:0/949704501 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6f80100950 con 0x7f6f88105580 2024-04-25T01:33:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 -- start start 2024-04-25T01:33:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f700a2c30 0x7f6f700a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee3774270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3779340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee377be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aee36d7cb0 con 0x55aee3773e50 2024-04-25T01:33:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aee36805b0 con 0x55aee3774840 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8fa28740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55aee36d3c70 con 0x55aee37799b0 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f86d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee377be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee3774270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f86d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee377be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:47744/0 (socket says 172.21.15.16:47744) 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f8659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee3774270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35454/0 (socket says 172.21.15.16:35454) 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.733+0000 7f6f86d9d640 1 -- 172.21.15.16:0/186431686 learned_addr learned my addr 172.21.15.16:0/186431686 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f86d9d640 1 -- 172.21.15.16:0/186431686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 msgr2=0x55aee3779340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3779340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f86d9d640 1 -- 172.21.15.16:0/186431686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 msgr2=0x55aee3774270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee3774270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f86d9d640 1 -- 172.21.15.16:0/186431686 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aee33ec210 con 0x55aee37799b0 2024-04-25T01:33:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f700a2c30 0x7f6f700a50f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6f8819f370 tx=0x7f6f7c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.734+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee377be40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f800cce90 tx=0x7f6f800ccaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8200f34010ebaad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.735+0000 7f6f74ff9640 1 -- 172.21.15.16:0/186431686 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8005b070 con 0x55aee37799b0 2024-04-25T01:33:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.735+0000 7f6f74ff9640 1 -- 172.21.15.16:0/186431686 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f800d1540 con 0x55aee37799b0 2024-04-25T01:33:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.735+0000 7f6f74ff9640 1 -- 172.21.15.16:0/186431686 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f800c9800 con 0x55aee37799b0 2024-04-25T01:33:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.736+0000 7f6f8fa28740 1 -- 172.21.15.16:0/186431686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 msgr2=0x55aee377be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.736+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee377be40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f800cce90 tx=0x7f6f800ccaf0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.737+0000 7f6f8fa28740 1 -- 172.21.15.16:0/186431686 shutdown_connections 2024-04-25T01:33:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.737+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee377be40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.737+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3779340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.737+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/186431686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee3774270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.737+0000 7f6f8fa28740 1 -- 172.21.15.16:0/186431686 >> 172.21.15.16:0/186431686 conn(0x55aee376f4c0 msgr2=0x55aee376f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.737+0000 7f6f8fa28740 1 -- 172.21.15.16:0/186431686 shutdown_connections 2024-04-25T01:33:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.738+0000 7f6f8fa28740 1 -- 172.21.15.16:0/186431686 wait complete. 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.738+0000 7f6f8fa28740 1 -- start start 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.738+0000 7f6f8fa28740 1 Processor -- start 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3773e50 0x55aee377e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee377e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3785600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee377e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee377e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60544/0 (socket says 172.21.15.16:60544) 2024-04-25T01:33:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f85d9b640 1 -- 172.21.15.16:0/2935492825 learned_addr learned my addr 172.21.15.16:0/2935492825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3773e50 0x55aee377e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.739+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55aee377c720 con 0x55aee37799b0 2024-04-25T01:33:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aee3693b50 con 0x55aee3774840 2024-04-25T01:33:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55aee377c8a0 con 0x55aee3773e50 2024-04-25T01:33:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8fa28740 10 client.0 _ll_register_callbacks cb 0x55aee36ceca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8659c640 1 -- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 msgr2=0x55aee377e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee377e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8659c640 1 -- 172.21.15.16:0/2935492825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 msgr2=0x55aee3785600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3785600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8659c640 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aee34b8620 con 0x55aee3773e50 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3773e50 0x55aee377e080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55aee37750a0 tx=0x7f6f78012a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=739938f852736027 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.740+0000 7f6f6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.741+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2935492825 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78004030 con 0x55aee3773e50 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.741+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f800d1540 con 0x55aee3773e50 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.741+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2935492825 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6f780041d0 con 0x55aee3773e50 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.741+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2935492825 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6f780043d0 con 0x55aee3773e50 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.742+0000 7f6f8fa28740 10 client.25351 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.742+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55aee3788a90 con 0x55aee3773e50 2024-04-25T01:33:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.742+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2935492825 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f800d1540 con 0x55aee3773e50 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2935492825 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6f7800f7c0 con 0x55aee3773e50 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f8fa28740 10 client.25351 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f8fa28740 10 client.25351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f8fa28740 20 client.25351 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee378a5f0 0x55aee378cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f8fa28740 4 client.25351 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aee36e51f0 con 0x55aee378a5f0 2024-04-25T01:33:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.743+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee378a5f0 0x55aee378cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.746+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee378a5f0 0x55aee378cab0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6df83d0a50ffbd95 server_cookie=58a72022a5c294a7 in_seq=0 out_seq=0 2024-04-25T01:33:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.746+0000 7f6f6dffb640 10 client.25351 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.746+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2935492825 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aee36e51f0 con 0x55aee378a5f0 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.746+0000 7f6f6dffb640 10 client.25351 handle_command_reply: tid=0 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.746+0000 7f6f8fa28740 1 client.25351 shutdown 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3773e50 msgr2=0x55aee377e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3773e50 0x55aee377e080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55aee37750a0 tx=0x7f6f78012a80 comp rx=0 tx=0).stop 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 shutdown_connections 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3785600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee378a5f0 0x55aee378cab0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee377e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2935492825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3773e50 0x55aee377e080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 >> 172.21.15.16:0/2935492825 conn(0x55aee376f4c0 msgr2=0x55aee376e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f6dffb640 0 client.25351 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 shutdown_connections 2024-04-25T01:33:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2935492825 wait complete. 2024-04-25T01:33:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.747+0000 7f6f8fa28740 20 client.25351 trim_cache size 0 max 16384 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 Processor -- start 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 -- start start 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee376dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3785520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee376e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88198470 con 0x55aee3773e50 2024-04-25T01:33:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88195d40 con 0x55aee3774840 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f8fa28740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88072930 con 0x55aee37799b0 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3785520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3785520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60546/0 (socket says 172.21.15.16:60546) 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f85d9b640 1 -- 172.21.15.16:0/2298312142 learned_addr learned my addr 172.21.15.16:0/2298312142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.775+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee376e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee376dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f85d9b640 1 -- 172.21.15.16:0/2298312142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 msgr2=0x55aee376e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee376e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f85d9b640 1 -- 172.21.15.16:0/2298312142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 msgr2=0x55aee376dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee376dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f85d9b640 1 -- 172.21.15.16:0/2298312142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aee37888b0 con 0x55aee3774840 2024-04-25T01:33:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee376dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3785520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c01d390 tx=0x7f6f7c012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04ef5167e40701b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f6ffff640 1 -- 172.21.15.16:0/2298312142 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c029070 con 0x55aee3774840 2024-04-25T01:33:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f6ffff640 1 -- 172.21.15.16:0/2298312142 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f7c01d3c0 con 0x55aee3774840 2024-04-25T01:33:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.776+0000 7f6f6ffff640 1 -- 172.21.15.16:0/2298312142 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c016070 con 0x55aee3774840 2024-04-25T01:33:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.777+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2298312142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 msgr2=0x55aee3785520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.777+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3785520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c01d390 tx=0x7f6f7c012000 comp rx=0 tx=0).stop 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.777+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2298312142 shutdown_connections 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.777+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee37799b0 0x55aee376e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3774840 0x55aee3785520 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2298312142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee3773e50 0x55aee376dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2298312142 >> 172.21.15.16:0/2298312142 conn(0x55aee376f4c0 msgr2=0x55aee376f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2298312142 shutdown_connections 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2298312142 wait complete. 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 -- start start 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.778+0000 7f6f8fa28740 1 Processor -- start 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 0x55aee3785240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3774840 0x55aee3781d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8fa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3782270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 0x55aee3785240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 0x55aee3785240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60560/0 (socket says 172.21.15.16:60560) 2024-04-25T01:33:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3774840 0x55aee3781d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f85d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3774840 0x55aee3781d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:47782/0 (socket says 172.21.15.16:47782) 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8659c640 1 -- 172.21.15.16:0/2741290380 learned_addr learned my addr 172.21.15.16:0/2741290380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88072fa0 con 0x55aee37799b0 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88195670 con 0x55aee3773e50 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.779+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88073670 con 0x55aee3774840 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8fa28740 10 client.0 _ll_register_callbacks cb 0x7f6f78009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f86d9d640 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3782270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8659c640 1 -- 172.21.15.16:0/2741290380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3774840 msgr2=0x55aee3781d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3774840 0x55aee3781d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8659c640 1 -- 172.21.15.16:0/2741290380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 msgr2=0x55aee3782270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3782270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8659c640 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aee3785870 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f8659c640 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 0x55aee3785240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6f78000c00 tx=0x7f6f7801c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d6bf74fe0c5259c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2741290380 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78035060 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.780+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2741290380 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6f78009d60 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aee36df540 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2741290380 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6f7802c040 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 10 client.25394 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55aee37efb20 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2741290380 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f7800fe50 con 0x55aee3773e50 2024-04-25T01:33:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2741290380 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6f780104c0 con 0x55aee3773e50 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 10 client.25394 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 10 client.25394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 20 client.25394 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee37f0500 0x55aee37f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 4 client.25394 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aee36df540 con 0x55aee37f0500 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.781+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee37f0500 0x55aee37f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.782+0000 7f6f85d9b640 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee37f0500 0x55aee37f2920 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57260d19a9b57782 server_cookie=4cb2aa1d98b39a4c in_seq=0 out_seq=0 2024-04-25T01:33:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.782+0000 7f6f6dffb640 10 client.25394 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f6dffb640 1 -- 172.21.15.16:0/2741290380 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aee36df540 con 0x55aee37f0500 2024-04-25T01:33:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f6dffb640 10 client.25394 handle_command_reply: tid=0 2024-04-25T01:33:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 client.25394 shutdown 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 msgr2=0x55aee3785240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 0x55aee3785240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6f78000c00 tx=0x7f6f7801c460 comp rx=0 tx=0).stop 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 shutdown_connections 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55aee37799b0 0x55aee3782270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aee3774840 0x55aee3781d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55aee37f0500 0x55aee37f2920 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 --2- 172.21.15.16:0/2741290380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aee3773e50 0x55aee3785240 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6f78000c00 tx=0x7f6f7801c460 comp rx=0 tx=0).stop 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 >> 172.21.15.16:0/2741290380 conn(0x55aee376f4c0 msgr2=0x55aee376dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.788+0000 7f6f6dffb640 0 client.25394 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.789+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 shutdown_connections 2024-04-25T01:33:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.789+0000 7f6f8fa28740 1 -- 172.21.15.16:0/2741290380 wait complete. 2024-04-25T01:33:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.789+0000 7f6f8fa28740 20 client.25394 trim_cache size 0 max 16384 2024-04-25T01:33:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:33:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:33:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f700a2c30 msgr2=0x7f6f700a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f700a2c30 0x7f6f700a50f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6f8819f370 tx=0x7f6f7c00a380 comp rx=0 tx=0).stop 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 msgr2=0x7f6f881a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f881a5410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f80034040 tx=0x7f6f80030fc0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 shutdown_connections 2024-04-25T01:33:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f6f700a2c30 0x7f6f700a50f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f88105580 0x7f6f881a5410 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f88104bc0 0x7f6f8819e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 --2- 172.21.15.16:0/949704501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f881039d0 0x7f6f8819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.790+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 >> 172.21.15.16:0/949704501 conn(0x7f6f880fef60 msgr2=0x7f6f88077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.791+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 shutdown_connections 2024-04-25T01:33:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:08.792+0000 7f6f8cc38640 1 -- 172.21.15.16:0/949704501 wait complete. 2024-04-25T01:33:09.748 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:09 smithi156 ceph-mon[32631]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:33:09.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:09 smithi016 ceph-mon[29153]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:33:09.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:09 smithi038 ceph-mon[32841]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T01:33:11.748 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:11 smithi156 ceph-mon[32631]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:33:11.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:11 smithi016 ceph-mon[29153]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:33:11.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:11 smithi038 ceph-mon[32841]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:33:13.748 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:13 smithi156 ceph-mon[32631]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:13.764 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:13 smithi038 ceph-mon[32841]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:13.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:13 smithi016 ceph-mon[29153]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:15.748 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:15 smithi156 ceph-mon[32631]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:33:15.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:15 smithi016 ceph-mon[29153]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:33:15.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:15 smithi038 ceph-mon[32841]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T01:33:17.747 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:17 smithi156 ceph-mon[32631]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 62 MiB/s wr, 1k op/s 2024-04-25T01:33:17.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:17 smithi016 ceph-mon[29153]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 62 MiB/s wr, 1k op/s 2024-04-25T01:33:17.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:17 smithi038 ceph-mon[32841]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 62 MiB/s wr, 1k op/s 2024-04-25T01:33:19.747 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:19 smithi156 ceph-mon[32631]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T01:33:19.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:19 smithi016 ceph-mon[29153]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T01:33:19.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:19 smithi038 ceph-mon[32841]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T01:33:21.747 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:21 smithi156 ceph-mon[32631]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:21.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:21 smithi016 ceph-mon[29153]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 -- 172.21.15.16:0/514391567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 msgr2=0x7fee9c105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 --2- 172.21.15.16:0/514391567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c105030 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fee98009900 tx=0x7fee9802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 -- 172.21.15.16:0/514391567 shutdown_connections 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 --2- 172.21.15.16:0/514391567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c105570 0x7fee9c10be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 --2- 172.21.15.16:0/514391567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c105030 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 --2- 172.21.15.16:0/514391567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c1039c0 0x7fee9c103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 -- 172.21.15.16:0/514391567 >> 172.21.15.16:0/514391567 conn(0x7fee9c0fef30 msgr2=0x7fee9c101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 -- 172.21.15.16:0/514391567 shutdown_connections 2024-04-25T01:33:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.956+0000 7feea4161640 1 -- 172.21.15.16:0/514391567 wait complete. 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 Processor -- start 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 -- start start 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c1039c0 0x7fee9c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c105570 0x7fee9c1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c10f520 con 0x7fee9c104bb0 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c10f3a0 con 0x7fee9c105570 2024-04-25T01:33:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea4161640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c10f6a0 con 0x7fee9c1039c0 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea16fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea1efd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c1039c0 0x7fee9c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea16fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c19a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58664/0 (socket says 172.21.15.16:58664) 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.957+0000 7feea16fc640 1 -- 172.21.15.16:0/346581960 learned_addr learned my addr 172.21.15.16:0/346581960 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea26fe640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c105570 0x7fee9c1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea1efd640 1 -- 172.21.15.16:0/346581960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c105570 msgr2=0x7fee9c1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea1efd640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c105570 0x7fee9c1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea1efd640 1 -- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 msgr2=0x7fee9c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea1efd640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea1efd640 1 -- 172.21.15.16:0/346581960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9c1a1750 con 0x7fee9c1039c0 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea26fe640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c105570 0x7fee9c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea16fc640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7feea1efd640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c1039c0 0x7fee9c199b30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e9c0 tx=0x7fee8c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.958+0000 7fee92ffd640 1 -- 172.21.15.16:0/346581960 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c00ccc0 con 0x7fee9c1039c0 2024-04-25T01:33:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.959+0000 7fee92ffd640 1 -- 172.21.15.16:0/346581960 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee8c0035f0 con 0x7fee9c1039c0 2024-04-25T01:33:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.959+0000 7fee92ffd640 1 -- 172.21.15.16:0/346581960 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c00ccc0 con 0x7fee9c1039c0 2024-04-25T01:33:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.959+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9c075be0 con 0x7fee9c1039c0 2024-04-25T01:33:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.959+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9c0760e0 con 0x7fee9c1039c0 2024-04-25T01:33:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.963+0000 7feea6e6c740 1 Processor -- start 2024-04-25T01:33:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.963+0000 7feea6e6c740 1 -- start start 2024-04-25T01:33:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf09270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf09840 0x5557caf0e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf0e9b0 0x5557caf10e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea6e6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557cae6ccb0 con 0x5557caf09840 2024-04-25T01:33:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea6e6c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5557cae155b0 con 0x5557caf08e50 2024-04-25T01:33:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea6e6c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5557cae68c70 con 0x5557caf0e9b0 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea26fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf09270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea26fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf09270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54184/0 (socket says 172.21.15.16:54184) 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7feea26fe640 1 -- 172.21.15.16:0/3522162519 learned_addr learned my addr 172.21.15.16:0/3522162519 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.964+0000 7fee92ffd640 1 -- 172.21.15.16:0/346581960 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fee8c0037a0 con 0x7fee9c1039c0 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea16fc640 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf0e9b0 0x5557caf10e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea26fe640 1 -- 172.21.15.16:0/3522162519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf0e9b0 msgr2=0x5557caf10e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea26fe640 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf0e9b0 0x5557caf10e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.943 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea26fe640 1 -- 172.21.15.16:0/3522162519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf09840 msgr2=0x5557caf0e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea1efd640 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf09840 0x5557caf0e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea26fe640 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf09840 0x5557caf0e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea26fe640 1 -- 172.21.15.16:0/3522162519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557cab81210 con 0x5557caf08e50 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea1efd640 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf09840 0x5557caf0e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.965+0000 7feea26fe640 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf09270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee94009fd0 tx=0x7fee940108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed8b1e2adacb10a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.966+0000 7fee87fff640 1 -- 172.21.15.16:0/3522162519 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94011760 con 0x5557caf08e50 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.966+0000 7fee92ffd640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fee880a2e40 0x7fee880a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.966+0000 7fee87fff640 1 -- 172.21.15.16:0/3522162519 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee94011db0 con 0x5557caf08e50 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.966+0000 7fee87fff640 1 -- 172.21.15.16:0/3522162519 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9400e3c0 con 0x5557caf08e50 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.966+0000 7fee92ffd640 1 -- 172.21.15.16:0/346581960 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fee8c0cf180 con 0x7fee9c1039c0 2024-04-25T01:33:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.966+0000 7feea1efd640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fee880a2e40 0x7fee880a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 -- 172.21.15.16:0/3522162519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 msgr2=0x5557caf09270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea1efd640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fee880a2e40 0x7fee880a5300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0ccfd0 tx=0x7fee8c003d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf09270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee94009fd0 tx=0x7fee940108e0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 -- 172.21.15.16:0/3522162519 shutdown_connections 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf0e9b0 0x5557caf10e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf09840 0x5557caf0e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3522162519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf09270 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 -- 172.21.15.16:0/3522162519 >> 172.21.15.16:0/3522162519 conn(0x5557caf044c0 msgr2=0x5557caf048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 -- 172.21.15.16:0/3522162519 shutdown_connections 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.967+0000 7feea6e6c740 1 -- 172.21.15.16:0/3522162519 wait complete. 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 -- start start 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 Processor -- start 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf13810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf1a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557caf11960 con 0x5557caf0e9b0 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5557cae28b50 con 0x5557caf08e50 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.968+0000 7feea6e6c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5557caf11ae0 con 0x5557caf09840 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.969+0000 7feea6e6c740 10 client.0 _ll_register_callbacks cb 0x5557cae63ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.969+0000 7feea16fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf13810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.969+0000 7feea16fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf13810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:55684/0 (socket says 172.21.15.16:55684) 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.969+0000 7feea16fc640 1 -- 172.21.15.16:0/1509740258 learned_addr learned my addr 172.21.15.16:0/1509740258 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.969+0000 7feea26fe640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf1a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.969+0000 7feea26fe640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.970+0000 7feea26fe640 1 -- 172.21.15.16:0/1509740258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 msgr2=0x5557caf13810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.970+0000 7feea26fe640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf13810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.970+0000 7feea26fe640 1 -- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 msgr2=0x5557caf132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.970+0000 7feea26fe640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf132d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.970+0000 7feea26fe640 1 -- 172.21.15.16:0/1509740258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557cac4d620 con 0x5557caf0e9b0 2024-04-25T01:33:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.970+0000 7feea26fe640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf1a850 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fee94000f80 tx=0x7fee94013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a3e17d1bc4df4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.972+0000 7fee857fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:33:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.972+0000 7fee857fa640 1 -- 172.21.15.16:0/1509740258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9402f060 con 0x5557caf0e9b0 2024-04-25T01:33:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.972+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee94011db0 con 0x5557caf0e9b0 2024-04-25T01:33:21.950 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.972+0000 7fee857fa640 1 -- 172.21.15.16:0/1509740258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fee940119a0 con 0x5557caf0e9b0 2024-04-25T01:33:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.973+0000 7fee857fa640 1 -- 172.21.15.16:0/1509740258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fee94011ba0 con 0x5557caf0e9b0 2024-04-25T01:33:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.973+0000 7feea6e6c740 10 client.15723 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.973+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5557caf1db30 con 0x5557caf0e9b0 2024-04-25T01:33:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.973+0000 7fee857fa640 1 -- 172.21.15.16:0/1509740258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee94011db0 con 0x5557caf0e9b0 2024-04-25T01:33:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7fee857fa640 1 -- 172.21.15.16:0/1509740258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fee94029040 con 0x5557caf0e9b0 2024-04-25T01:33:21.951 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7feea6e6c740 10 client.15723 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:21.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7feea6e6c740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:21.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7feea6e6c740 20 client.15723 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:21.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7feea6e6c740 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf1efe0 0x5557caf213d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7feea6e6c740 4 client.15723 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:21.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.974+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557cae7a1f0 con 0x5557caf1efe0 2024-04-25T01:33:21.952 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.975+0000 7feea16fc640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf1efe0 0x5557caf213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.976+0000 7feea16fc640 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf1efe0 0x5557caf213d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c586924af73a60 server_cookie=f437ed453698f33c in_seq=0 out_seq=0 2024-04-25T01:33:21.953 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.976+0000 7fee857fa640 10 client.15723 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:21.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.977+0000 7fee857fa640 1 -- 172.21.15.16:0/1509740258 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557cae7a1f0 con 0x5557caf1efe0 2024-04-25T01:33:21.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.977+0000 7fee857fa640 10 client.15723 handle_command_reply: tid=0 2024-04-25T01:33:21.954 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.977+0000 7feea6e6c740 1 client.15723 shutdown 2024-04-25T01:33:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 msgr2=0x5557caf1a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf1a850 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fee94000f80 tx=0x7fee94013040 comp rx=0 tx=0).stop 2024-04-25T01:33:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 shutdown_connections 2024-04-25T01:33:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf1a850 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf13810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.955 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf1efe0 0x5557caf213d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 --2- 172.21.15.16:0/1509740258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 >> 172.21.15.16:0/1509740258 conn(0x5557caf044c0 msgr2=0x5557caf035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7fee857fa640 0 client.15723 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 shutdown_connections 2024-04-25T01:33:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.978+0000 7feea6e6c740 1 -- 172.21.15.16:0/1509740258 wait complete. 2024-04-25T01:33:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:21.979+0000 7feea6e6c740 20 client.15723 trim_cache size 0 max 16384 2024-04-25T01:33:21.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:21 smithi038 ceph-mon[32841]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:21.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.004+0000 7feea6e6c740 1 Processor -- start 2024-04-25T01:33:21.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.004+0000 7feea6e6c740 1 -- start start 2024-04-25T01:33:21.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 0x5557caf02d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf1a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf0e9b0 0x5557caf03270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea26fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 0x5557caf02d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea6e6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1a9470 con 0x5557caf08e50 2024-04-25T01:33:21.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea6e6c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1a6d40 con 0x5557caf0e9b0 2024-04-25T01:33:21.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea6e6c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1a4bb0 con 0x5557caf09840 2024-04-25T01:33:21.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea26fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 0x5557caf02d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58690/0 (socket says 172.21.15.16:58690) 2024-04-25T01:33:21.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.005+0000 7feea26fe640 1 -- 172.21.15.16:0/3742683388 learned_addr learned my addr 172.21.15.16:0/3742683388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf0e9b0 0x5557caf03270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea16fc640 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf1a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 -- 172.21.15.16:0/3742683388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 msgr2=0x5557caf1a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf1a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 -- 172.21.15.16:0/3742683388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf0e9b0 msgr2=0x5557caf03270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf0e9b0 0x5557caf03270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 -- 172.21.15.16:0/3742683388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557caf04060 con 0x5557caf08e50 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.006+0000 7feea26fe640 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 0x5557caf02d30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fee94006010 tx=0x7fee940021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5802e806ed9407e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.007+0000 7fee877fe640 1 -- 172.21.15.16:0/3742683388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9402f070 con 0x5557caf08e50 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.007+0000 7fee877fe640 1 -- 172.21.15.16:0/3742683388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee9401cdc0 con 0x5557caf08e50 2024-04-25T01:33:21.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.007+0000 7fee877fe640 1 -- 172.21.15.16:0/3742683388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94035b10 con 0x5557caf08e50 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 -- 172.21.15.16:0/3742683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 msgr2=0x5557caf02d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 0x5557caf02d30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fee94006010 tx=0x7fee940021d0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 -- 172.21.15.16:0/3742683388 shutdown_connections 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf0e9b0 0x5557caf03270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf1a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 --2- 172.21.15.16:0/3742683388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf08e50 0x5557caf02d30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.008+0000 7feea6e6c740 1 -- 172.21.15.16:0/3742683388 >> 172.21.15.16:0/3742683388 conn(0x5557caf044c0 msgr2=0x5557caf0aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.009+0000 7feea6e6c740 1 -- 172.21.15.16:0/3742683388 shutdown_connections 2024-04-25T01:33:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.009+0000 7feea6e6c740 1 -- 172.21.15.16:0/3742683388 wait complete. 2024-04-25T01:33:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.009+0000 7feea6e6c740 1 -- start start 2024-04-25T01:33:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 Processor -- start 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf1a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf17350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf17980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea26fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf1a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1a5200 con 0x5557caf0e9b0 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea26fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf1a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54210/0 (socket says 172.21.15.16:54210) 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1a6670 con 0x5557caf08e50 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea26fe640 1 -- 172.21.15.16:0/2521260493 learned_addr learned my addr 172.21.15.16:0/2521260493 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.010+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c1a58d0 con 0x5557caf09840 2024-04-25T01:33:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea6e6c740 10 client.0 _ll_register_callbacks cb 0x7fee94009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea16fc640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf17350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea26fe640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf17980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea26fe640 1 -- 172.21.15.16:0/2521260493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 msgr2=0x5557caf17350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea26fe640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf17350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea26fe640 1 -- 172.21.15.16:0/2521260493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 msgr2=0x5557caf17980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea26fe640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf17980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea26fe640 1 -- 172.21.15.16:0/2521260493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557caf83580 con 0x5557caf08e50 2024-04-25T01:33:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.011+0000 7feea16fc640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf17350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.012+0000 7feea26fe640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf1a8b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fee94005fe0 tx=0x7fee94005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b584a59d5924c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.012+0000 7fee857fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:33:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.013+0000 7fee857fa640 1 -- 172.21.15.16:0/2521260493 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9402f040 con 0x5557caf08e50 2024-04-25T01:33:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.013+0000 7fee857fa640 1 -- 172.21.15.16:0/2521260493 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fee9401ba70 con 0x5557caf08e50 2024-04-25T01:33:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.013+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee9401cdc0 con 0x5557caf08e50 2024-04-25T01:33:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.013+0000 7fee857fa640 1 -- 172.21.15.16:0/2521260493 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fee9401bc70 con 0x5557caf08e50 2024-04-25T01:33:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.013+0000 7feea6e6c740 10 client.25412 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:21.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.013+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5557caf84890 con 0x5557caf08e50 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7fee857fa640 1 -- 172.21.15.16:0/2521260493 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee9401cdc0 con 0x5557caf08e50 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7fee857fa640 1 -- 172.21.15.16:0/2521260493 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fee94046490 con 0x5557caf08e50 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea6e6c740 10 client.25412 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea6e6c740 10 client.25412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea6e6c740 20 client.25412 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea6e6c740 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf850a0 0x5557caf874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea6e6c740 4 client.25412 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557cae74540 con 0x5557caf850a0 2024-04-25T01:33:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.014+0000 7feea16fc640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf850a0 0x5557caf874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.015+0000 7fee857fa640 10 client.25412 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.015+0000 7feea16fc640 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf850a0 0x5557caf874c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ebe4ae14355780f server_cookie=ffbec4a88bcbf15 in_seq=0 out_seq=0 2024-04-25T01:33:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.022+0000 7fee857fa640 1 -- 172.21.15.16:0/2521260493 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557cae74540 con 0x5557caf850a0 2024-04-25T01:33:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.022+0000 7fee857fa640 10 client.25412 handle_command_reply: tid=0 2024-04-25T01:33:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.022+0000 7feea6e6c740 1 client.25412 shutdown 2024-04-25T01:33:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 msgr2=0x5557caf1a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf1a8b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fee94005fe0 tx=0x7fee94005f70 comp rx=0 tx=0).stop 2024-04-25T01:33:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 shutdown_connections 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557caf0e9b0 0x5557caf17980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557caf09840 0x5557caf17350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5557caf850a0 0x5557caf874c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 --2- 172.21.15.16:0/2521260493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557caf08e50 0x5557caf1a8b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7fee857fa640 0 client.25412 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.023+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 >> 172.21.15.16:0/2521260493 conn(0x5557caf044c0 msgr2=0x5557caf02d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.024+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 shutdown_connections 2024-04-25T01:33:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.024+0000 7feea6e6c740 1 -- 172.21.15.16:0/2521260493 wait complete. 2024-04-25T01:33:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.024+0000 7feea6e6c740 20 client.25412 trim_cache size 0 max 16384 2024-04-25T01:33:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:33:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:33:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:33:22.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:33:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.027+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fee880a2e40 msgr2=0x7fee880a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.027+0000 7feea4161640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fee880a2e40 0x7fee880a5300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0ccfd0 tx=0x7fee8c003d00 comp rx=0 tx=0).stop 2024-04-25T01:33:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.027+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c1039c0 msgr2=0x7fee9c199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.027+0000 7feea4161640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c1039c0 0x7fee9c199b30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e9c0 tx=0x7fee8c00ee80 comp rx=0 tx=0).stop 2024-04-25T01:33:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.028+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 shutdown_connections 2024-04-25T01:33:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.028+0000 7feea4161640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fee880a2e40 0x7fee880a5300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.028+0000 7feea4161640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee9c105570 0x7fee9c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.028+0000 7feea4161640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee9c104bb0 0x7fee9c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.028+0000 7feea4161640 1 --2- 172.21.15.16:0/346581960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9c1039c0 0x7fee9c199b30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.028+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 >> 172.21.15.16:0/346581960 conn(0x7fee9c0fef30 msgr2=0x7fee9c107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.029+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 shutdown_connections 2024-04-25T01:33:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:22.029+0000 7feea4161640 1 -- 172.21.15.16:0/346581960 wait complete. 2024-04-25T01:33:23.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:23 smithi016 ceph-mon[29153]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:23.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:23 smithi038 ceph-mon[32841]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:23.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:23 smithi156 ceph-mon[32631]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:25.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:25 smithi016 ceph-mon[29153]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:33:25.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:25 smithi038 ceph-mon[32841]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:33:25.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:25 smithi156 ceph-mon[32631]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T01:33:27.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:27 smithi016 ceph-mon[29153]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-04-25T01:33:27.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:27 smithi038 ceph-mon[32841]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-04-25T01:33:27.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:27 smithi156 ceph-mon[32631]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-04-25T01:33:29.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:29 smithi016 ceph-mon[29153]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:29.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:29 smithi038 ceph-mon[32841]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:29.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:29 smithi156 ceph-mon[32631]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:31.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:31 smithi016 ceph-mon[29153]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:31.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:31 smithi038 ceph-mon[32841]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:32.001 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:31 smithi156 ceph-mon[32631]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:33.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:33 smithi016 ceph-mon[29153]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:33.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:33 smithi038 ceph-mon[32841]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:33.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:33 smithi156 ceph-mon[32631]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 -- 172.21.15.16:0/2643644812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e0104bc0 msgr2=0x7f55e0105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.162 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 --2- 172.21.15.16:0/2643644812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e0104bc0 0x7f55e0105040 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f55d0009900 tx=0x7f55d002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 -- 172.21.15.16:0/2643644812 shutdown_connections 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 --2- 172.21.15.16:0/2643644812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55e0105580 0x7f55e010be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 --2- 172.21.15.16:0/2643644812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e0104bc0 0x7f55e0105040 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 --2- 172.21.15.16:0/2643644812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55e01039d0 0x7f55e0103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.185+0000 7f55e5e58640 1 -- 172.21.15.16:0/2643644812 >> 172.21.15.16:0/2643644812 conn(0x7f55e00fef60 msgr2=0x7f55e0101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.186+0000 7f55e5e58640 1 -- 172.21.15.16:0/2643644812 shutdown_connections 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.186+0000 7f55e5e58640 1 -- 172.21.15.16:0/2643644812 wait complete. 2024-04-25T01:33:35.163 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 Processor -- start 2024-04-25T01:33:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 -- start start 2024-04-25T01:33:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 0x7f55e0070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55e0104bc0 0x7f55e00706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55e0105580 0x7f55e01ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.164 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55e010f4e0 con 0x7f55e0104bc0 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55e010f360 con 0x7f55e01039d0 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55e5e58640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55e010f660 con 0x7f55e0105580 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.187+0000 7f55dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55e0105580 0x7f55e01ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 0x7f55e0070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 0x7f55e0070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:36382/0 (socket says 172.21.15.16:36382) 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 -- 172.21.15.16:0/3594183427 learned_addr learned my addr 172.21.15.16:0/3594183427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:35.165 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55deffd640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55e0104bc0 0x7f55e00706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 -- 172.21.15.16:0/3594183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55e0105580 msgr2=0x7f55e01ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55e0105580 0x7f55e01ae3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 -- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55e0104bc0 msgr2=0x7f55e00706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55e0104bc0 0x7f55e00706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 -- 172.21.15.16:0/3594183427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e0079ba0 con 0x7f55e01039d0 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55df7fe640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 0x7f55e0070160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e9c0 tx=0x7f55d400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.188+0000 7f55dcff9640 1 -- 172.21.15.16:0/3594183427 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d400cd80 con 0x7f55e01039d0 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.189+0000 7f55dcff9640 1 -- 172.21.15.16:0/3594183427 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d40108e0 con 0x7f55e01039d0 2024-04-25T01:33:35.166 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.189+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e01a5bd0 con 0x7f55e01039d0 2024-04-25T01:33:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.189+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e01a6170 con 0x7f55e01039d0 2024-04-25T01:33:35.167 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.189+0000 7f55dcff9640 1 -- 172.21.15.16:0/3594183427 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4018840 con 0x7f55e01039d0 2024-04-25T01:33:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 Processor -- start 2024-04-25T01:33:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 -- start start 2024-04-25T01:33:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55dcff9640 1 -- 172.21.15.16:0/3594183427 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f55d401f020 con 0x7f55e01039d0 2024-04-25T01:33:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b015710270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b015710840 0x55b015715340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 0x55b015717e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b015673cb0 con 0x55b01570fe50 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b01561c5b0 con 0x55b0157159b0 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.192+0000 7f55e8b65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b01566fc70 con 0x55b015710840 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 0x55b015717e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b015710270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 0x55b015717e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:36390/0 (socket says 172.21.15.16:36390) 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55dffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b015710270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56256/0 (socket says 172.21.15.16:56256) 2024-04-25T01:33:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 -- 172.21.15.16:0/695689529 learned_addr learned my addr 172.21.15.16:0/695689529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55df7fe640 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b015710840 0x55b015715340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 -- 172.21.15.16:0/695689529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b015710840 msgr2=0x55b015715340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b015710840 0x55b015715340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 -- 172.21.15.16:0/695689529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 msgr2=0x55b015710270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b015710270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 -- 172.21.15.16:0/695689529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b015388210 con 0x55b0157159b0 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55dcff9640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55c80a2cb0 0x7f55c80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.193+0000 7f55deffd640 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 0x55b015717e40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55d002a910 tx=0x7f55d0032b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93bd1568f69f4e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.194+0000 7f55c5ffb640 1 -- 172.21.15.16:0/695689529 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d003f070 con 0x55b0157159b0 2024-04-25T01:33:35.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.194+0000 7f55dcff9640 1 -- 172.21.15.16:0/3594183427 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55d40cf1d0 con 0x7f55e01039d0 2024-04-25T01:33:35.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.194+0000 7f55c5ffb640 1 -- 172.21.15.16:0/695689529 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d0004030 con 0x55b0157159b0 2024-04-25T01:33:35.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.194+0000 7f55c5ffb640 1 -- 172.21.15.16:0/695689529 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0035070 con 0x55b0157159b0 2024-04-25T01:33:35.174 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.194+0000 7f55dffff640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55c80a2cb0 0x7f55c80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 -- 172.21.15.16:0/695689529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 msgr2=0x55b015717e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55dffff640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55c80a2cb0 0x7f55c80a5170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55b0157110a0 tx=0x7f55d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 0x55b015717e40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55d002a910 tx=0x7f55d0032b00 comp rx=0 tx=0).stop 2024-04-25T01:33:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 -- 172.21.15.16:0/695689529 shutdown_connections 2024-04-25T01:33:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0157159b0 0x55b015717e40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b015710840 0x55b015715340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 --2- 172.21.15.16:0/695689529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b015710270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 -- 172.21.15.16:0/695689529 >> 172.21.15.16:0/695689529 conn(0x55b01570b4c0 msgr2=0x55b01570b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 -- 172.21.15.16:0/695689529 shutdown_connections 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 -- 172.21.15.16:0/695689529 wait complete. 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.195+0000 7f55e8b65740 1 -- start start 2024-04-25T01:33:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 Processor -- start 2024-04-25T01:33:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b01571a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b015710840 0x55b01571a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b015718900 con 0x55b01570fe50 2024-04-25T01:33:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b01562fb50 con 0x55b015710840 2024-04-25T01:33:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b015718a80 con 0x55b0157159b0 2024-04-25T01:33:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55e8b65740 10 client.0 _ll_register_callbacks cb 0x55b01566aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b01571a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b01571a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56260/0 (socket says 172.21.15.16:56260) 2024-04-25T01:33:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55df7fe640 1 -- 172.21.15.16:0/850612622 learned_addr learned my addr 172.21.15.16:0/850612622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55dffff640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55deffd640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b015710840 0x55b01571a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55df7fe640 1 -- 172.21.15.16:0/850612622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 msgr2=0x55b0157217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.196+0000 7f55df7fe640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157217d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55df7fe640 1 -- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b015710840 msgr2=0x55b01571a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55df7fe640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b015710840 0x55b01571a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.183 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55df7fe640 1 -- 172.21.15.16:0/850612622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b015454620 con 0x55b01570fe50 2024-04-25T01:33:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55df7fe640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b01571a250 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f55d409b9a0 tx=0x7f55d409b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c24098673719b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55beffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:33:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55beffd640 1 -- 172.21.15.16:0/850612622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d40ce070 con 0x55b01570fe50 2024-04-25T01:33:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55beffd640 1 -- 172.21.15.16:0/850612622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55d40a41f0 con 0x55b01570fe50 2024-04-25T01:33:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55d0004030 con 0x55b01570fe50 2024-04-25T01:33:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55beffd640 1 -- 172.21.15.16:0/850612622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f55d40d2e60 con 0x55b01570fe50 2024-04-25T01:33:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55e8b65740 10 client.15741 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.197+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b015724c10 con 0x55b01570fe50 2024-04-25T01:33:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55beffd640 1 -- 172.21.15.16:0/850612622 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d0004030 con 0x55b01570fe50 2024-04-25T01:33:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55beffd640 1 -- 172.21.15.16:0/850612622 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f55d409ae60 con 0x55b01570fe50 2024-04-25T01:33:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55e8b65740 10 client.15741 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55e8b65740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55e8b65740 20 client.15741 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55e8b65740 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b015728d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55e8b65740 4 client.15741 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0156811f0 con 0x55b015726890 2024-04-25T01:33:35.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.198+0000 7f55deffd640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b015728d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.199+0000 7f55deffd640 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b015728d50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eedb67ce518e7456 server_cookie=64fe9089758dd416 in_seq=0 out_seq=0 2024-04-25T01:33:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.199+0000 7f55beffd640 10 client.15741 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.199+0000 7f55beffd640 1 -- 172.21.15.16:0/850612622 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0156811f0 con 0x55b015726890 2024-04-25T01:33:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.199+0000 7f55beffd640 10 client.15741 handle_command_reply: tid=0 2024-04-25T01:33:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.199+0000 7f55e8b65740 1 client.15741 shutdown 2024-04-25T01:33:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 msgr2=0x55b01571a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b01571a250 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f55d409b9a0 tx=0x7f55d409b8a0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 shutdown_connections 2024-04-25T01:33:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157217d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b015728d50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b015710840 0x55b01571a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 --2- 172.21.15.16:0/850612622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01570fe50 0x55b01571a250 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55beffd640 0 client.15741 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:35.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 >> 172.21.15.16:0/850612622 conn(0x55b01570b4c0 msgr2=0x55b01570a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 shutdown_connections 2024-04-25T01:33:35.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 1 -- 172.21.15.16:0/850612622 wait complete. 2024-04-25T01:33:35.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.200+0000 7f55e8b65740 20 client.15741 trim_cache size 0 max 16384 2024-04-25T01:33:35.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.231+0000 7f55e8b65740 1 Processor -- start 2024-04-25T01:33:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.232+0000 7f55e8b65740 1 -- start start 2024-04-25T01:33:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.233+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01570b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.233+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b015721760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.233+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b015709d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55e8b65740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc010d90 con 0x55b015710840 2024-04-25T01:33:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55e8b65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc00e660 con 0x55b01570fe50 2024-04-25T01:33:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55e8b65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc00c4b0 con 0x55b0157159b0 2024-04-25T01:33:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b015709d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b015709d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:50922/0 (socket says 172.21.15.16:50922) 2024-04-25T01:33:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55dffff640 1 -- 172.21.15.16:0/3861736877 learned_addr learned my addr 172.21.15.16:0/3861736877 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.234+0000 7f55deffd640 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b015721760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55deffd640 1 -- 172.21.15.16:0/3861736877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 msgr2=0x55b015709d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55deffd640 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b015709d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55deffd640 1 -- 172.21.15.16:0/3861736877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 msgr2=0x55b01570b8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:33:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55deffd640 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01570b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55deffd640 1 -- 172.21.15.16:0/3861736877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01570b060 con 0x55b015710840 2024-04-25T01:33:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55deffd640 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b015721760 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f55d0000c00 tx=0x7f55d0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f224947ab6978d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.235+0000 7f55c57fa640 1 -- 172.21.15.16:0/3861736877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d003f040 con 0x55b015710840 2024-04-25T01:33:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.236+0000 7f55c57fa640 1 -- 172.21.15.16:0/3861736877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d0031780 con 0x55b015710840 2024-04-25T01:33:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.236+0000 7f55c57fa640 1 -- 172.21.15.16:0/3861736877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0035040 con 0x55b015710840 2024-04-25T01:33:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 -- 172.21.15.16:0/3861736877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 msgr2=0x55b015721760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b015721760 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f55d0000c00 tx=0x7f55d0005a20 comp rx=0 tx=0).stop 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 -- 172.21.15.16:0/3861736877 shutdown_connections 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b015709d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b015721760 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 --2- 172.21.15.16:0/3861736877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01570b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 -- 172.21.15.16:0/3861736877 >> 172.21.15.16:0/3861736877 conn(0x55b01570b4c0 msgr2=0x55b015711240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 -- 172.21.15.16:0/3861736877 shutdown_connections 2024-04-25T01:33:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 -- 172.21.15.16:0/3861736877 wait complete. 2024-04-25T01:33:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.237+0000 7f55e8b65740 1 -- start start 2024-04-25T01:33:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 Processor -- start 2024-04-25T01:33:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01571ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b01571f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157201d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01571ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc00cb20 con 0x55b015710840 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc00df90 con 0x55b01570fe50 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01571ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:36416/0 (socket says 172.21.15.16:36416) 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55e8b65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc00d1f0 con 0x55b0157159b0 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.238+0000 7f55df7fe640 1 -- 172.21.15.16:0/334159441 learned_addr learned my addr 172.21.15.16:0/334159441 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55e8b65740 10 client.0 _ll_register_callbacks cb 0x7f55d40d29a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55deffd640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b01571f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55dffff640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157201d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55df7fe640 1 -- 172.21.15.16:0/334159441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 msgr2=0x55b0157201d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55df7fe640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157201d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55df7fe640 1 -- 172.21.15.16:0/334159441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 msgr2=0x55b01571f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55df7fe640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b01571f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55df7fe640 1 -- 172.21.15.16:0/334159441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b015715700 con 0x55b01570fe50 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55df7fe640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01571ed60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55d40058c0 tx=0x7f55d401fff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1701c7600f8a1b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55beffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55beffd640 1 -- 172.21.15.16:0/334159441 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d40ce070 con 0x55b01570fe50 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55beffd640 1 -- 172.21.15.16:0/334159441 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55d4010630 con 0x55b01570fe50 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.239+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55d0031780 con 0x55b01570fe50 2024-04-25T01:33:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55beffd640 1 -- 172.21.15.16:0/334159441 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f55d4097cb0 con 0x55b01570fe50 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 10 client.25427 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b0157218d0 con 0x55b01570fe50 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55beffd640 1 -- 172.21.15.16:0/334159441 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d4009470 con 0x55b01570fe50 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55beffd640 1 -- 172.21.15.16:0/334159441 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f55d40ba2a0 con 0x55b01570fe50 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 10 client.25427 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 10 client.25427 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 20 client.25427 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b01578f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 4 client.25427 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f55d0031780 con 0x55b015726890 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.240+0000 7f55deffd640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b01578f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.241+0000 7f55beffd640 10 client.25427 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.241+0000 7f55deffd640 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b01578f320 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c17222aad2be9bb3 server_cookie=854a6f651e873676 in_seq=0 out_seq=0 2024-04-25T01:33:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.244+0000 7f55beffd640 1 -- 172.21.15.16:0/334159441 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f55d0031780 con 0x55b015726890 2024-04-25T01:33:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.244+0000 7f55beffd640 10 client.25427 handle_command_reply: tid=0 2024-04-25T01:33:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.244+0000 7f55e8b65740 1 client.25427 shutdown 2024-04-25T01:33:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 msgr2=0x55b01571ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01571ed60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55d40058c0 tx=0x7f55d401fff0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 shutdown_connections 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0157159b0 0x55b0157201d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b015710840 0x55b01571f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55b015726890 0x55b01578f320 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 --2- 172.21.15.16:0/334159441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b01570fe50 0x55b01571ed60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55beffd640 0 client.25427 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 >> 172.21.15.16:0/334159441 conn(0x55b01570b4c0 msgr2=0x55b01570b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 shutdown_connections 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 1 -- 172.21.15.16:0/334159441 wait complete. 2024-04-25T01:33:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.245+0000 7f55e8b65740 20 client.25427 trim_cache size 0 max 16384 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55c80a2cb0 msgr2=0x7f55c80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55c80a2cb0 0x7f55c80a5170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55b0157110a0 tx=0x7f55d8009500 comp rx=0 tx=0).stop 2024-04-25T01:33:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 msgr2=0x7f55e0070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 0x7f55e0070160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e9c0 tx=0x7f55d400edc0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 shutdown_connections 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f55c80a2cb0 0x7f55c80a5170 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55e0105580 0x7f55e01ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55e0104bc0 0x7f55e00706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 --2- 172.21.15.16:0/3594183427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55e01039d0 0x7f55e0070160 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 >> 172.21.15.16:0/3594183427 conn(0x7f55e00fef60 msgr2=0x7f55e0107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 shutdown_connections 2024-04-25T01:33:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:35.246+0000 7f55e5e58640 1 -- 172.21.15.16:0/3594183427 wait complete. 2024-04-25T01:33:35.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:35 smithi016 ceph-mon[29153]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:33:35.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:35 smithi038 ceph-mon[32841]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:33:35.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:35 smithi156 ceph-mon[32631]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-25T01:33:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:37 smithi016 ceph-mon[29153]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-04-25T01:33:37.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:37 smithi038 ceph-mon[32841]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-04-25T01:33:37.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:37 smithi156 ceph-mon[32631]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 985 op/s 2024-04-25T01:33:39.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:39 smithi016 ceph-mon[29153]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:39.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:39 smithi038 ceph-mon[32841]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:39.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:39 smithi156 ceph-mon[32631]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:41.799 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:41 smithi156 ceph-mon[32631]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:41.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:41 smithi016 ceph-mon[29153]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:41.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:41 smithi038 ceph-mon[32841]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T01:33:43.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:43 smithi016 ceph-mon[29153]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:33:43.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:43 smithi038 ceph-mon[32841]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:33:43.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:43 smithi156 ceph-mon[32631]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-04-25T01:33:45.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:45 smithi016 ceph-mon[29153]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T01:33:45.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:45 smithi038 ceph-mon[32841]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T01:33:45.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:45 smithi156 ceph-mon[32631]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T01:33:47.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:47 smithi016 ceph-mon[29153]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 61 MiB/s wr, 979 op/s 2024-04-25T01:33:47.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:47 smithi038 ceph-mon[32841]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 61 MiB/s wr, 979 op/s 2024-04-25T01:33:47.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:47 smithi156 ceph-mon[32631]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 61 MiB/s wr, 979 op/s 2024-04-25T01:33:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 -- 172.21.15.16:0/3540050155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0108d50 msgr2=0x7fb1c01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/3540050155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0108d50 0x7fb1c01091d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009860 tx=0x7fb1b002a360 comp rx=0 tx=0).stop 2024-04-25T01:33:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 -- 172.21.15.16:0/3540050155 shutdown_connections 2024-04-25T01:33:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/3540050155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 0x7fb1c010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/3540050155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0108d50 0x7fb1c01091d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/3540050155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0072a70 0x7fb1c0072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 -- 172.21.15.16:0/3540050155 >> 172.21.15.16:0/3540050155 conn(0x7fb1c006e5e0 msgr2=0x7fb1c0070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 -- 172.21.15.16:0/3540050155 shutdown_connections 2024-04-25T01:33:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.410+0000 7fb1c6beb640 1 -- 172.21.15.16:0/3540050155 wait complete. 2024-04-25T01:33:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.411+0000 7fb1c6beb640 1 Processor -- start 2024-04-25T01:33:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.411+0000 7fb1c6beb640 1 -- start start 2024-04-25T01:33:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.411+0000 7fb1c6beb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0072a70 0x7fb1c01a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.411+0000 7fb1c6beb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0108d50 0x7fb1c01a6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.411+0000 7fb1c6beb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 0x7fb1c01adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.411+0000 7fb1c6beb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c011bf80 con 0x7fb1c0108d50 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c6beb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c011be00 con 0x7fb1c0109710 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c6beb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c011c100 con 0x7fb1c0072a70 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c63ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 0x7fb1c01adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c5be9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0072a70 0x7fb1c01a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c5be9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0072a70 0x7fb1c01a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40704/0 (socket says 172.21.15.16:40704) 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c5be9640 1 -- 172.21.15.16:0/1478620552 learned_addr learned my addr 172.21.15.16:0/1478620552 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.412+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0108d50 0x7fb1c01a6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c53e8640 1 -- 172.21.15.16:0/1478620552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0072a70 msgr2=0x7fb1c01a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0072a70 0x7fb1c01a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c53e8640 1 -- 172.21.15.16:0/1478620552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 msgr2=0x7fb1c01adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 0x7fb1c01adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c53e8640 1 -- 172.21.15.16:0/1478620552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c010ffa0 con 0x7fb1c0108d50 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0108d50 0x7fb1c01a6bf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009780 tx=0x7fb1b0032830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.413+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 0x7fb1c01adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.414+0000 7fb1b6ffd640 1 -- 172.21.15.16:0/1478620552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0037040 con 0x7fb1c0108d50 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.414+0000 7fb1b6ffd640 1 -- 172.21.15.16:0/1478620552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1b0032c20 con 0x7fb1c0108d50 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.414+0000 7fb1b6ffd640 1 -- 172.21.15.16:0/1478620552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0003850 con 0x7fb1c0108d50 2024-04-25T01:33:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.414+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c0110230 con 0x7fb1c0108d50 2024-04-25T01:33:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.414+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c0110640 con 0x7fb1c0108d50 2024-04-25T01:33:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.417+0000 7fb1cac38740 1 Processor -- start 2024-04-25T01:33:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 -- start start 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x556677560450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x556677565520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677568020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5566774c3cb0 con 0x556677560030 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55667746c5b0 con 0x556677560a20 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1cac38740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5566774bfc70 con 0x556677565b90 2024-04-25T01:33:48.395 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.418+0000 7fb1b6ffd640 1 -- 172.21.15.16:0/1478620552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fb1b0043020 con 0x7fb1c0108d50 2024-04-25T01:33:48.396 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x556677565520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x556677565520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46352/0 (socket says 172.21.15.16:46352) 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 -- 172.21.15.16:0/2627153162 learned_addr learned my addr 172.21.15.16:0/2627153162 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677568020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x556677560450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 -- 172.21.15.16:0/2627153162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 msgr2=0x556677568020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677568020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 -- 172.21.15.16:0/2627153162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 msgr2=0x556677560450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x556677560450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 -- 172.21.15.16:0/2627153162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566771d8210 con 0x556677560a20 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x556677560450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.419+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x556677565520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800e2b0 tx=0x7fb1b8009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbac8bb00b5f452a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.420+0000 7fb1abfff640 1 -- 172.21.15.16:0/2627153162 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8018020 con 0x556677560a20 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.420+0000 7fb1b6ffd640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1ac0a2c00 0x7fb1ac0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.420+0000 7fb1abfff640 1 -- 172.21.15.16:0/2627153162 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1b8014d40 con 0x556677560a20 2024-04-25T01:33:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.420+0000 7fb1b6ffd640 1 -- 172.21.15.16:0/1478620552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb1b00ee5c0 con 0x7fb1c0108d50 2024-04-25T01:33:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.420+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1ac0a2c00 0x7fb1ac0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.420+0000 7fb1abfff640 1 -- 172.21.15.16:0/2627153162 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8013da0 con 0x556677560a20 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 -- 172.21.15.16:0/2627153162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 msgr2=0x556677565520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x556677565520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800e2b0 tx=0x7fb1b8009770 comp rx=0 tx=0).stop 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1ac0a2c00 0x7fb1ac0a50c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x556677561280 tx=0x7fb1bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 -- 172.21.15.16:0/2627153162 shutdown_connections 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677568020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x556677565520 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2627153162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x556677560450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 -- 172.21.15.16:0/2627153162 >> 172.21.15.16:0/2627153162 conn(0x55667755b6a0 msgr2=0x55667755baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 -- 172.21.15.16:0/2627153162 shutdown_connections 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 -- 172.21.15.16:0/2627153162 wait complete. 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 -- start start 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.421+0000 7fb1cac38740 1 Processor -- start 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x55667756a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x55667756a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x5566775717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1c5be9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x55667756a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1c5be9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x55667756a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42486/0 (socket says 172.21.15.16:42486) 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1c5be9640 1 -- 172.21.15.16:0/2495149145 learned_addr learned my addr 172.21.15.16:0/2495149145 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556677568be0 con 0x556677560030 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55667747fb50 con 0x556677560a20 2024-04-25T01:33:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556677568d60 con 0x556677565b90 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.422+0000 7fb1cac38740 10 client.0 _ll_register_callbacks cb 0x5566774baca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x55667756a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x5566775717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 -- 172.21.15.16:0/2495149145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 msgr2=0x5566775717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x5566775717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 -- 172.21.15.16:0/2495149145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 msgr2=0x55667756a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x55667756a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566772a4620 con 0x556677560a20 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x55667756a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x55667756a780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005e00 tx=0x7fb1b0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=369d491545dddc09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.423+0000 7fb1a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1a97fa640 1 -- 172.21.15.16:0/2495149145 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00bab70 con 0x556677560a20 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1a97fa640 1 -- 172.21.15.16:0/2495149145 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb1b00c3580 con 0x556677560a20 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b8014d40 con 0x556677560a20 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1a97fa640 1 -- 172.21.15.16:0/2495149145 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb1b00ba700 con 0x556677560a20 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1cac38740 10 client.25439 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556677574aa0 con 0x556677560a20 2024-04-25T01:33:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1a97fa640 1 -- 172.21.15.16:0/2495149145 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1b8014d40 con 0x556677560a20 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1a97fa640 1 -- 172.21.15.16:0/2495149145 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb1b00ee0d0 con 0x556677560a20 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1cac38740 10 client.25439 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.424+0000 7fb1cac38740 10 client.25439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1cac38740 20 client.25439 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x556677578650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1cac38740 4 client.25439 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566774d11f0 con 0x5566775761b0 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x556677578650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1a97fa640 10 client.25439 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.425+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x556677578650 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a521ed9898666e28 server_cookie=775d3d893772e8ee in_seq=0 out_seq=0 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.426+0000 7fb1a97fa640 1 -- 172.21.15.16:0/2495149145 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566774d11f0 con 0x5566775761b0 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.426+0000 7fb1a97fa640 10 client.25439 handle_command_reply: tid=0 2024-04-25T01:33:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.427+0000 7fb1cac38740 1 client.25439 shutdown 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.427+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 msgr2=0x55667756a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.427+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x55667756a780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005e00 tx=0x7fb1b0005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 shutdown_connections 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x5566775717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x556677578650 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1a97fa640 0 client.25439 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560a20 0x55667756a780 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 --2- 172.21.15.16:0/2495149145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560030 0x55667756a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 >> 172.21.15.16:0/2495149145 conn(0x55667755b6a0 msgr2=0x55667755a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.428+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 shutdown_connections 2024-04-25T01:33:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.429+0000 7fb1cac38740 1 -- 172.21.15.16:0/2495149145 wait complete. 2024-04-25T01:33:48.408 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.429+0000 7fb1cac38740 20 client.25439 trim_cache size 0 max 16384 2024-04-25T01:33:48.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.463+0000 7fb1cac38740 1 Processor -- start 2024-04-25T01:33:48.440 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.463+0000 7fb1cac38740 1 -- start start 2024-04-25T01:33:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667755e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677560a20 0x556677571780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677565b90 0x55667755e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1c53e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677560a20 0x556677571780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1c53e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677560a20 0x556677571780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40718/0 (socket says 172.21.15.16:40718) 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1c53e8640 1 -- 172.21.15.16:0/975483109 learned_addr learned my addr 172.21.15.16:0/975483109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c01c29d0 con 0x556677565b90 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c01c02a0 con 0x556677560030 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.464+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667755e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.465+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677565b90 0x55667755e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.465+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c01175b0 con 0x556677560a20 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.466+0000 7fb1c5be9640 1 -- 172.21.15.16:0/975483109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677560a20 msgr2=0x556677571780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.466+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677560a20 0x556677571780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.466+0000 7fb1c5be9640 1 -- 172.21.15.16:0/975483109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677565b90 msgr2=0x55667755e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.466+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677565b90 0x55667755e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.466+0000 7fb1c5be9640 1 -- 172.21.15.16:0/975483109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556677564dd0 con 0x556677560030 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.466+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667755e420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b010 tx=0x7fb1b8004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aac2b140fd06f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.467+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677565b90 0x55667755e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.467+0000 7fb1ab7fe640 1 -- 172.21.15.16:0/975483109 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8018040 con 0x556677560030 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.467+0000 7fb1ab7fe640 1 -- 172.21.15.16:0/975483109 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1b8014690 con 0x556677560030 2024-04-25T01:33:48.444 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.467+0000 7fb1ab7fe640 1 -- 172.21.15.16:0/975483109 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8009450 con 0x556677560030 2024-04-25T01:33:48.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 msgr2=0x55667755e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667755e420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b010 tx=0x7fb1b8004ac0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 shutdown_connections 2024-04-25T01:33:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677565b90 0x55667755e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677560a20 0x556677571780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 --2- 172.21.15.16:0/975483109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667755e420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 >> 172.21.15.16:0/975483109 conn(0x55667755b6a0 msgr2=0x556677561b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 shutdown_connections 2024-04-25T01:33:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.469+0000 7fb1cac38740 1 -- 172.21.15.16:0/975483109 wait complete. 2024-04-25T01:33:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.470+0000 7fb1cac38740 1 -- start start 2024-04-25T01:33:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.470+0000 7fb1cac38740 1 Processor -- start 2024-04-25T01:33:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.470+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667756eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560a20 0x55667756f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1cac38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677570020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1cac38740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c0117c20 con 0x556677560a20 2024-04-25T01:33:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1cac38740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c01bfbd0 con 0x556677560030 2024-04-25T01:33:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1cac38740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c01182f0 con 0x556677565b90 2024-04-25T01:33:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1cac38740 10 client.0 _ll_register_callbacks cb 0x7fb1b0005780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1c63ea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677570020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1c63ea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677570020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40724/0 (socket says 172.21.15.16:40724) 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1c63ea640 1 -- 172.21.15.16:0/774268949 learned_addr learned my addr 172.21.15.16:0/774268949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.471+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667756eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c53e8640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560a20 0x55667756f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c63ea640 1 -- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 msgr2=0x55667756eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667756eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c63ea640 1 -- 172.21.15.16:0/774268949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560a20 msgr2=0x55667756f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560a20 0x55667756f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c63ea640 1 -- 172.21.15.16:0/774268949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556677570610 con 0x556677565b90 2024-04-25T01:33:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.472+0000 7fb1c63ea640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677570020 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc011bd0 tx=0x7fb1bc015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f42c989166fade8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:33:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1a97fa640 1 -- 172.21.15.16:0/774268949 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc02d060 con 0x556677565b90 2024-04-25T01:33:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1a97fa640 1 -- 172.21.15.16:0/774268949 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb1bc028600 con 0x556677565b90 2024-04-25T01:33:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b8014690 con 0x556677565b90 2024-04-25T01:33:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1a97fa640 1 -- 172.21.15.16:0/774268949 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb1bc028800 con 0x556677565b90 2024-04-25T01:33:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1cac38740 10 client.25411 fetch_fsmap learned FSMap version 22 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.473+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556677571960 con 0x556677565b90 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1a97fa640 1 -- 172.21.15.16:0/774268949 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1b8014690 con 0x556677565b90 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1a97fa640 1 -- 172.21.15.16:0/774268949 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb1bc033a40 con 0x556677565b90 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1cac38740 10 client.25411 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1cac38740 10 client.25411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1cac38740 20 client.25411 populate_metadata read hostname 'smithi016' 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1cac38740 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x5566775deea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1cac38740 4 client.25411 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566774cb540 con 0x5566775761b0 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.474+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x5566775deea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.475+0000 7fb1a97fa640 10 client.25411 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.476+0000 7fb1c5be9640 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x5566775deea0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e88582cf889db29 server_cookie=2ca872521b31b2cf in_seq=0 out_seq=0 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.484+0000 7fb1a97fa640 1 -- 172.21.15.16:0/774268949 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566774cb540 con 0x5566775761b0 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.484+0000 7fb1a97fa640 10 client.25411 handle_command_reply: tid=0 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.484+0000 7fb1cac38740 1 client.25411 shutdown 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 msgr2=0x556677570020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677570020 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc011bd0 tx=0x7fb1bc015040 comp rx=0 tx=0).stop 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 shutdown_connections 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556677565b90 0x556677570020 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556677560a20 0x55667756f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5566775761b0 0x5566775deea0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 --2- 172.21.15.16:0/774268949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556677560030 0x55667756eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1a97fa640 0 client.25411 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 >> 172.21.15.16:0/774268949 conn(0x55667755b6a0 msgr2=0x55667755a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 shutdown_connections 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 1 -- 172.21.15.16:0/774268949 wait complete. 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.485+0000 7fb1cac38740 20 client.25411 trim_cache size 0 max 16384 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:33:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:33:48.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:33:48.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1ac0a2c00 msgr2=0x7fb1ac0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.464 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1ac0a2c00 0x7fb1ac0a50c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x556677561280 tx=0x7fb1bc009260 comp rx=0 tx=0).stop 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0108d50 msgr2=0x7fb1c01a6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0108d50 0x7fb1c01a6bf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009780 tx=0x7fb1b0032830 comp rx=0 tx=0).stop 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 shutdown_connections 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fb1ac0a2c00 0x7fb1ac0a50c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1c0109710 0x7fb1c01adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1c0108d50 0x7fb1c01a6bf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 --2- 172.21.15.16:0/1478620552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1c0072a70 0x7fb1c01a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 >> 172.21.15.16:0/1478620552 conn(0x7fb1c006e5e0 msgr2=0x7fb1c006efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 shutdown_connections 2024-04-25T01:33:48.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:33:48.486+0000 7fb1c6beb640 1 -- 172.21.15.16:0/1478620552 wait complete. 2024-04-25T01:33:48.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:48 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:48.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:48 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:48.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:48 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:49.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:49 smithi016 ceph-mon[29153]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:49.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:49 smithi038 ceph-mon[32841]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:49.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:49 smithi156 ceph-mon[32631]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T01:33:50.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:50.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:50.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:50.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:33:50.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:50.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:33:51.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:51 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-25T01:33:51.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:51 smithi016 ceph-mon[29153]: Unable to set osd_memory_target on smithi016 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:33:51.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:51 smithi016 ceph-mon[29153]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:51.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:51 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-25T01:33:51.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:51 smithi038 ceph-mon[32841]: Unable to set osd_memory_target on smithi016 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:33:51.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:51 smithi038 ceph-mon[32841]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:51.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:51 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-25T01:33:51.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:51 smithi156 ceph-mon[32631]: Unable to set osd_memory_target on smithi016 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:33:51.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:51 smithi156 ceph-mon[32631]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:53.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:53 smithi016 ceph-mon[29153]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:33:53.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:53 smithi038 ceph-mon[32841]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:33:53.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:53 smithi156 ceph-mon[32631]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-25T01:33:55.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:55 smithi016 ceph-mon[29153]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:55.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:55 smithi038 ceph-mon[32841]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:55.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:55 smithi156 ceph-mon[32631]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-04-25T01:33:57.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:57 smithi016 ceph-mon[29153]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 59 MiB/s wr, 950 op/s 2024-04-25T01:33:57.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:57 smithi038 ceph-mon[32841]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 59 MiB/s wr, 950 op/s 2024-04-25T01:33:57.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:57 smithi156 ceph-mon[32631]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 59 MiB/s wr, 950 op/s 2024-04-25T01:33:59.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:33:59 smithi016 ceph-mon[29153]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:59.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:33:59 smithi038 ceph-mon[32841]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:33:59.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:33:59 smithi156 ceph-mon[32631]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-25T01:34:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 -- 172.21.15.16:0/3825831170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0108fc0 msgr2=0x7f91f0109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 --2- 172.21.15.16:0/3825831170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0108fc0 0x7f91f0109440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009900 tx=0x7f91e002a410 comp rx=0 tx=0).stop 2024-04-25T01:34:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 -- 172.21.15.16:0/3825831170 shutdown_connections 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 --2- 172.21.15.16:0/3825831170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f0110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 --2- 172.21.15.16:0/3825831170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0108fc0 0x7f91f0109440 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 --2- 172.21.15.16:0/3825831170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0107dd0 0x7f91f01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 -- 172.21.15.16:0/3825831170 >> 172.21.15.16:0/3825831170 conn(0x7f91f0076d90 msgr2=0x7f91f00791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.658+0000 7f91f58dd640 1 -- 172.21.15.16:0/3825831170 shutdown_connections 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.660+0000 7f91f58dd640 1 -- 172.21.15.16:0/3825831170 wait complete. 2024-04-25T01:34:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.661+0000 7f91f58dd640 1 Processor -- start 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 -- start start 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0107dd0 0x7f91f0070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0108fc0 0x7f91f00705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f01ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0113980 con 0x7f91f0108fc0 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0113800 con 0x7f91f0107dd0 2024-04-25T01:34:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91f58dd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0113b00 con 0x7f91f0109980 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91ef7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f01ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91ee7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0108fc0 0x7f91f00705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91ef7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f01ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35780/0 (socket says 172.21.15.16:35780) 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91ee7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0108fc0 0x7f91f00705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38420/0 (socket says 172.21.15.16:38420) 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.662+0000 7f91ef7fe640 1 -- 172.21.15.16:0/2777075236 learned_addr learned my addr 172.21.15.16:0/2777075236 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91ef7fe640 1 -- 172.21.15.16:0/2777075236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0107dd0 msgr2=0x7f91f0070080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0107dd0 0x7f91f0070080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91ef7fe640 1 -- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0108fc0 msgr2=0x7f91f00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0108fc0 0x7f91f00705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91ef7fe640 1 -- 172.21.15.16:0/2777075236 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f0070c60 con 0x7f91f0109980 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f01ae090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f91e400b3f0 tx=0x7f91e400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91f48db640 1 -- 172.21.15.16:0/2777075236 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4015020 con 0x7f91f0109980 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91f48db640 1 -- 172.21.15.16:0/2777075236 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91e4011ce0 con 0x7f91f0109980 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91f48db640 1 -- 172.21.15.16:0/2777075236 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4010da0 con 0x7f91f0109980 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f01ae6f0 con 0x7f91f0109980 2024-04-25T01:34:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.663+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f01aec90 con 0x7f91f0109980 2024-04-25T01:34:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.666+0000 7f91f48db640 1 -- 172.21.15.16:0/2777075236 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f91e4018020 con 0x7f91f0109980 2024-04-25T01:34:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.667+0000 7f91f85ea740 1 Processor -- start 2024-04-25T01:34:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.668+0000 7f91f48db640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d80a2d30 0x7f91d80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.668+0000 7f91f48db640 1 -- 172.21.15.16:0/2777075236 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f91e40ce8f0 con 0x7f91f0109980 2024-04-25T01:34:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 -- start start 2024-04-25T01:34:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714cc840 0x55f5714d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 0x55f5714d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.648 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f57142fcb0 con 0x55f5714cc840 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d80a2d30 0x7f91d80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f5713d85b0 con 0x55f5714d19b0 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.670+0000 7f91f85ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f57142bc70 con 0x55f5714cbe50 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 0x55f5714d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 0x55f5714d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54598/0 (socket says 172.21.15.16:54598) 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 -- 172.21.15.16:0/651292199 learned_addr learned my addr 172.21.15.16:0/651292199 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714cc840 0x55f5714d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 -- 172.21.15.16:0/651292199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 msgr2=0x55f5714cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 -- 172.21.15.16:0/651292199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714cc840 msgr2=0x55f5714d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714cc840 0x55f5714d1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ef7fe640 1 -- 172.21.15.16:0/651292199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f571144210 con 0x55f5714d19b0 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.671+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714cc840 0x55f5714d1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.672+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 0x55f5714d3e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91e40a3810 tx=0x7f91e409ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5ccfdcdb3253358 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:01.650 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.672+0000 7f91d5ffb640 1 -- 172.21.15.16:0/651292199 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e40b4070 con 0x55f5714d19b0 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.672+0000 7f91d5ffb640 1 -- 172.21.15.16:0/651292199 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91e409b0d0 con 0x55f5714d19b0 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.673+0000 7f91d5ffb640 1 -- 172.21.15.16:0/651292199 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e40bb0b0 con 0x55f5714d19b0 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.673+0000 7f91f85ea740 1 -- 172.21.15.16:0/651292199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 msgr2=0x55f5714d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.673+0000 7f91f85ea740 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 0x55f5714d3e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91e40a3810 tx=0x7f91e409ad10 comp rx=0 tx=0).stop 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.673+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d80a2d30 0x7f91d80a51f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f91e0006010 tx=0x7f91e0032f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 -- 172.21.15.16:0/651292199 shutdown_connections 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714d19b0 0x55f5714d3e40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714cc840 0x55f5714d1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 --2- 172.21.15.16:0/651292199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 -- 172.21.15.16:0/651292199 >> 172.21.15.16:0/651292199 conn(0x55f5714c74c0 msgr2=0x55f5714c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 -- 172.21.15.16:0/651292199 shutdown_connections 2024-04-25T01:34:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.674+0000 7f91f85ea740 1 -- 172.21.15.16:0/651292199 wait complete. 2024-04-25T01:34:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.675+0000 7f91f85ea740 1 -- start start 2024-04-25T01:34:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.675+0000 7f91f85ea740 1 Processor -- start 2024-04-25T01:34:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.675+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714d6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cc840 0x55f5714d66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714dd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91f85ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f5714d49b0 con 0x55f5714d19b0 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91f85ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f5713ebb50 con 0x55f5714cc840 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91f85ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f5714d4b30 con 0x55f5714cbe50 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91f85ea740 10 client.0 _ll_register_callbacks cb 0x55f571426ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714d6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91ee7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cc840 0x55f5714d66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714d6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35802/0 (socket says 172.21.15.16:35802) 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 -- 172.21.15.16:0/2245489454 learned_addr learned my addr 172.21.15.16:0/2245489454 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714dd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 -- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cc840 msgr2=0x55f5714d66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cc840 0x55f5714d66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 -- 172.21.15.16:0/2245489454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 msgr2=0x55f5714dd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714dd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91eeffd640 1 -- 172.21.15.16:0/2245489454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f571210620 con 0x55f5714cbe50 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cc840 0x55f5714d66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.676+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714dd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91eeffd640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714d6160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91e800e710 tx=0x7f91e800ebd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=464f7256fa7e780 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91ceffd640 1 -- 172.21.15.16:0/2245489454 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8004cb0 con 0x55f5714cbe50 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91ceffd640 1 -- 172.21.15.16:0/2245489454 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f91e800c7d0 con 0x55f5714cbe50 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91e409b0d0 con 0x55f5714cbe50 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91ceffd640 1 -- 172.21.15.16:0/2245489454 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f91e8004e50 con 0x55f5714cbe50 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91f85ea740 10 client.25417 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.677+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f5714e0b20 con 0x55f5714cbe50 2024-04-25T01:34:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91ceffd640 1 -- 172.21.15.16:0/2245489454 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91e409b0d0 con 0x55f5714cbe50 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91ceffd640 1 -- 172.21.15.16:0/2245489454 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f91e8010430 con 0x55f5714cbe50 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91f85ea740 10 client.25417 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91f85ea740 10 client.25417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91f85ea740 20 client.25417 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f5714e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91f85ea740 4 client.25417 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f57143d1f0 con 0x55f5714e2680 2024-04-25T01:34:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.678+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f5714e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.679+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f5714e4ae0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2105f394975e0b99 server_cookie=dcf1385d4d9596a6 in_seq=0 out_seq=0 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.679+0000 7f91ceffd640 10 client.25417 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.680+0000 7f91ceffd640 1 -- 172.21.15.16:0/2245489454 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f57143d1f0 con 0x55f5714e2680 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.680+0000 7f91ceffd640 10 client.25417 handle_command_reply: tid=0 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.680+0000 7f91f85ea740 1 client.25417 shutdown 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.681+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 msgr2=0x55f5714d6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.681+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714d6160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91e800e710 tx=0x7f91e800ebd0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 shutdown_connections 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714dd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f5714e4ae0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cc840 0x55f5714d66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91ceffd640 0 client.25417 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2245489454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cbe50 0x55f5714d6160 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 >> 172.21.15.16:0/2245489454 conn(0x55f5714c74c0 msgr2=0x55f5714c65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 shutdown_connections 2024-04-25T01:34:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 1 -- 172.21.15.16:0/2245489454 wait complete. 2024-04-25T01:34:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.682+0000 7f91f85ea740 20 client.25417 trim_cache size 0 max 16384 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 Processor -- start 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 -- start start 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714dd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714d5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01ba3e0 con 0x55f5714d19b0 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01b7cb0 con 0x55f5714cbe50 2024-04-25T01:34:01.686 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.709+0000 7f91f85ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01b5b00 con 0x55f5714cc840 2024-04-25T01:34:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54614/0 (socket says 172.21.15.16:54614) 2024-04-25T01:34:01.687 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 -- 172.21.15.16:0/960866924 learned_addr learned my addr 172.21.15.16:0/960866924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714d5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714dd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 -- 172.21.15.16:0/960866924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 msgr2=0x55f5714dd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714dd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 -- 172.21.15.16:0/960866924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 msgr2=0x55f5714d5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714d5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 -- 172.21.15.16:0/960866924 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5714ccc20 con 0x55f5714cbe50 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714d5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.710+0000 7f91eeffd640 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714caad0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f91e8005d70 tx=0x7f91e800e6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d740b50c58adb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:01.688 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.711+0000 7f91d57fa640 1 -- 172.21.15.16:0/960866924 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8030070 con 0x55f5714cbe50 2024-04-25T01:34:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.711+0000 7f91d57fa640 1 -- 172.21.15.16:0/960866924 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91e8010630 con 0x55f5714cbe50 2024-04-25T01:34:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.711+0000 7f91d57fa640 1 -- 172.21.15.16:0/960866924 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8028070 con 0x55f5714cbe50 2024-04-25T01:34:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.712+0000 7f91f85ea740 1 -- 172.21.15.16:0/960866924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 msgr2=0x55f5714caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.689 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.712+0000 7f91f85ea740 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714caad0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f91e8005d70 tx=0x7f91e800e6a0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 -- 172.21.15.16:0/960866924 shutdown_connections 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f5714d5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714dd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 --2- 172.21.15.16:0/960866924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714caad0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 -- 172.21.15.16:0/960866924 >> 172.21.15.16:0/960866924 conn(0x55f5714c74c0 msgr2=0x55f5714c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 -- 172.21.15.16:0/960866924 shutdown_connections 2024-04-25T01:34:01.690 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 -- 172.21.15.16:0/960866924 wait complete. 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.713+0000 7f91f85ea740 1 -- start start 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 Processor -- start 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714da250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714da790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f571546c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01b6170 con 0x55f5714d19b0 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01b75e0 con 0x55f5714cbe50 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01b6840 con 0x55f5714cc840 2024-04-25T01:34:01.691 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91f85ea740 10 client.0 _ll_register_callbacks cb 0x7f91e800e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:01.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714da250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.692 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714da250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:54616/0 (socket says 172.21.15.16:54616) 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.714+0000 7f91eeffd640 1 -- 172.21.15.16:0/2971928311 learned_addr learned my addr 172.21.15.16:0/2971928311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f571546c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714da790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91eeffd640 1 -- 172.21.15.16:0/2971928311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 msgr2=0x55f5714da790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91eeffd640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714da790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91eeffd640 1 -- 172.21.15.16:0/2971928311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 msgr2=0x55f571546c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91eeffd640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f571546c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91eeffd640 1 -- 172.21.15.16:0/2971928311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5714dd860 con 0x55f5714cbe50 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ef7fe640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f571546c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714da790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91eeffd640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714da250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f91e8010f80 tx=0x7f91e8009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbe960155da8012 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:01.693 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:34:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ceffd640 1 -- 172.21.15.16:0/2971928311 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8028070 con 0x55f5714cbe50 2024-04-25T01:34:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91ceffd640 1 -- 172.21.15.16:0/2971928311 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f91e8022420 con 0x55f5714cbe50 2024-04-25T01:34:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.715+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91e8010630 con 0x55f5714cbe50 2024-04-25T01:34:01.694 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.716+0000 7f91ceffd640 1 -- 172.21.15.16:0/2971928311 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f91e8030070 con 0x55f5714cbe50 2024-04-25T01:34:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.719+0000 7f91f85ea740 10 client.25460 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:01.695 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.719+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f571548030 con 0x55f5714cbe50 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91ceffd640 1 -- 172.21.15.16:0/2971928311 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91e8010630 con 0x55f5714cbe50 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91ceffd640 1 -- 172.21.15.16:0/2971928311 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f91e8018910 con 0x55f5714cbe50 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91f85ea740 10 client.25460 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91f85ea740 10 client.25460 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91f85ea740 20 client.25460 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f57154a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91f85ea740 4 client.25460 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:34:01.697 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.720+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f571437540 con 0x55f5714e2680 2024-04-25T01:34:01.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.722+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f57154a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.724+0000 7f91ee7fc640 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f57154a910 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4cfadc689f89c21 server_cookie=7fb66eff458a9f9a in_seq=0 out_seq=0 2024-04-25T01:34:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.724+0000 7f91ceffd640 10 client.25460 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.728+0000 7f91ceffd640 1 -- 172.21.15.16:0/2971928311 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f571437540 con 0x55f5714e2680 2024-04-25T01:34:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.728+0000 7f91ceffd640 10 client.25460 handle_command_reply: tid=0 2024-04-25T01:34:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.728+0000 7f91f85ea740 1 client.25460 shutdown 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 msgr2=0x55f5714da250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714da250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f91e8010f80 tx=0x7f91e8009fd0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 shutdown_connections 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f5714d19b0 0x55f571546c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5714cc840 0x55f5714da790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f5714e2680 0x55f57154a910 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91ceffd640 0 client.25460 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 --2- 172.21.15.16:0/2971928311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5714cbe50 0x55f5714da250 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 >> 172.21.15.16:0/2971928311 conn(0x55f5714c74c0 msgr2=0x55f5714caad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 shutdown_connections 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 1 -- 172.21.15.16:0/2971928311 wait complete. 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.729+0000 7f91f85ea740 20 client.25460 trim_cache size 0 max 16384 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:34:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:34:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d80a2d30 msgr2=0x7f91d80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d80a2d30 0x7f91d80a51f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f91e0006010 tx=0x7f91e0032f70 comp rx=0 tx=0).stop 2024-04-25T01:34:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 msgr2=0x7f91f01ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f01ae090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f91e400b3f0 tx=0x7f91e400b8b0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 shutdown_connections 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f91d80a2d30 0x7f91d80a51f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0109980 0x7f91f01ae090 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91f0108fc0 0x7f91f00705c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 --2- 172.21.15.16:0/2777075236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f0107dd0 0x7f91f0070080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.730+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 >> 172.21.15.16:0/2777075236 conn(0x7f91f0076d90 msgr2=0x7f91f010c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.731+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 shutdown_connections 2024-04-25T01:34:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:01.731+0000 7f91f58dd640 1 -- 172.21.15.16:0/2777075236 wait complete. 2024-04-25T01:34:01.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:01 smithi016 ceph-mon[29153]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:34:01.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:01 smithi038 ceph-mon[32841]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:34:01.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:01 smithi156 ceph-mon[32631]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T01:34:03.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:03 smithi016 ceph-mon[29153]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 66 MiB/s wr, 1.07k op/s 2024-04-25T01:34:03.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:03 smithi038 ceph-mon[32841]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 66 MiB/s wr, 1.07k op/s 2024-04-25T01:34:03.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:03 smithi156 ceph-mon[32631]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 66 MiB/s wr, 1.07k op/s 2024-04-25T01:34:05.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:05 smithi016 ceph-mon[29153]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:34:05.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:05 smithi038 ceph-mon[32841]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:34:05.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:05 smithi156 ceph-mon[32631]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T01:34:07.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:07 smithi016 ceph-mon[29153]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 60 MiB/s wr, 966 op/s 2024-04-25T01:34:07.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:07 smithi038 ceph-mon[32841]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 60 MiB/s wr, 966 op/s 2024-04-25T01:34:07.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:07 smithi156 ceph-mon[32631]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 60 MiB/s wr, 966 op/s 2024-04-25T01:34:09.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:09 smithi016 ceph-mon[29153]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:09.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:09 smithi038 ceph-mon[32841]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:09.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:09 smithi156 ceph-mon[32631]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:11.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:11 smithi016 ceph-mon[29153]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:11.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:11 smithi038 ceph-mon[32841]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:11.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:11 smithi156 ceph-mon[32631]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:12.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:12 smithi038 ceph-mon[32841]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 66 MiB/s wr, 1.08k op/s 2024-04-25T01:34:12.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:12 smithi156 ceph-mon[32631]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 66 MiB/s wr, 1.08k op/s 2024-04-25T01:34:13.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:12 smithi016 ceph-mon[29153]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 66 MiB/s wr, 1.08k op/s 2024-04-25T01:34:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 -- 172.21.15.16:0/2860722386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681027c0 msgr2=0x7f6268102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 --2- 172.21.15.16:0/2860722386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681027c0 0x7f6268102c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f625c009860 tx=0x7f625c02a340 comp rx=0 tx=0).stop 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 -- 172.21.15.16:0/2860722386 shutdown_connections 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 --2- 172.21.15.16:0/2860722386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 0x7f6268109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 --2- 172.21.15.16:0/2860722386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681027c0 0x7f6268102c40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 --2- 172.21.15.16:0/2860722386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62681015d0 0x7f62681019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.862+0000 7f626d6b9640 1 -- 172.21.15.16:0/2860722386 >> 172.21.15.16:0/2860722386 conn(0x7f62680fcb40 msgr2=0x7f62680fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.863+0000 7f626d6b9640 1 -- 172.21.15.16:0/2860722386 shutdown_connections 2024-04-25T01:34:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.863+0000 7f626d6b9640 1 -- 172.21.15.16:0/2860722386 wait complete. 2024-04-25T01:34:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.864+0000 7f626d6b9640 1 Processor -- start 2024-04-25T01:34:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.864+0000 7f626d6b9640 1 -- start start 2024-04-25T01:34:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.864+0000 7f626d6b9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 0x7f626819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626d6b9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62681027c0 0x7f626819e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626d6b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 0x7f62681a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626d6b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f626810d170 con 0x7f6268103180 2024-04-25T01:34:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626d6b9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f626810cff0 con 0x7f62681015d0 2024-04-25T01:34:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626d6b9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f626810d2f0 con 0x7f62681027c0 2024-04-25T01:34:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626ceb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 0x7f62681a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f6267fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 0x7f626819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626ceb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 0x7f62681a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45596/0 (socket says 172.21.15.16:45596) 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f626ceb8640 1 -- 172.21.15.16:0/347147012 learned_addr learned my addr 172.21.15.16:0/347147012 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f6267fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 0x7f626819dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:32778/0 (socket says 172.21.15.16:32778) 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.865+0000 7f62677fe640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62681027c0 0x7f626819e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f6267fff640 1 -- 172.21.15.16:0/347147012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62681027c0 msgr2=0x7f626819e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f6267fff640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62681027c0 0x7f626819e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f6267fff640 1 -- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 msgr2=0x7f62681a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f6267fff640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 0x7f62681a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f6267fff640 1 -- 172.21.15.16:0/347147012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62681a5900 con 0x7f62681015d0 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f6267fff640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 0x7f626819dcc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f625800b3c0 tx=0x7f625800b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f62657fa640 1 -- 172.21.15.16:0/347147012 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258011830 con 0x7f62681015d0 2024-04-25T01:34:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f62657fa640 1 -- 172.21.15.16:0/347147012 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6258016070 con 0x7f62681015d0 2024-04-25T01:34:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.866+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62681a5bf0 con 0x7f62681015d0 2024-04-25T01:34:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.867+0000 7f62657fa640 1 -- 172.21.15.16:0/347147012 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258010e30 con 0x7f62681015d0 2024-04-25T01:34:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.868+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62681a6100 con 0x7f62681015d0 2024-04-25T01:34:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.871+0000 7f627162c740 1 Processor -- start 2024-04-25T01:34:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.871+0000 7f62657fa640 1 -- 172.21.15.16:0/347147012 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f625803eb30 con 0x7f62681015d0 2024-04-25T01:34:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.871+0000 7f627162c740 1 -- start start 2024-04-25T01:34:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f627162c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ecfc0 0x55f9283ed3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f627162c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f627162c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f627162c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f928350cb0 con 0x55f9283ed9b0 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f627162c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f9282f95b0 con 0x55f9283ecfc0 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f627162c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f92834cc70 con 0x55f9283f2b20 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f626ceb8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ecfc0 0x55f9283ed3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f626ceb8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ecfc0 0x55f9283ed3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:32784/0 (socket says 172.21.15.16:32784) 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f626ceb8640 1 -- 172.21.15.16:0/2260128575 learned_addr learned my addr 172.21.15.16:0/2260128575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f6267fff640 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.872+0000 7f62677fe640 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f6267fff640 1 -- 172.21.15.16:0/2260128575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 msgr2=0x55f9283f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f6267fff640 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f6267fff640 1 -- 172.21.15.16:0/2260128575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ecfc0 msgr2=0x55f9283ed3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f6267fff640 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ecfc0 0x55f9283ed3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f6267fff640 1 -- 172.21.15.16:0/2260128575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f928065210 con 0x55f9283ed9b0 2024-04-25T01:34:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f62657fa640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f62500a2c80 0x7f62500a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f6267fff640 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f24b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f62580cb890 tx=0x7f62580d47c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf03522dea2a034 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f626ceb8640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f62500a2c80 0x7f62500a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.873+0000 7f62657fa640 1 -- 172.21.15.16:0/347147012 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f62580cde70 con 0x7f62681015d0 2024-04-25T01:34:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.874+0000 7f624e7fc640 1 -- 172.21.15.16:0/2260128575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62580968f0 con 0x55f9283ed9b0 2024-04-25T01:34:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.874+0000 7f624e7fc640 1 -- 172.21.15.16:0/2260128575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6258085080 con 0x55f9283ed9b0 2024-04-25T01:34:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.874+0000 7f624e7fc640 1 -- 172.21.15.16:0/2260128575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258099fb0 con 0x55f9283ed9b0 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f626ceb8640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f62500a2c80 0x7f62500a5140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6260009a60 tx=0x7f6260010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 -- 172.21.15.16:0/2260128575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 msgr2=0x55f9283f24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f24b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f62580cb890 tx=0x7f62580d47c0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 -- 172.21.15.16:0/2260128575 shutdown_connections 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f24b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 --2- 172.21.15.16:0/2260128575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ecfc0 0x55f9283ed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 -- 172.21.15.16:0/2260128575 >> 172.21.15.16:0/2260128575 conn(0x55f9283e8630 msgr2=0x55f9283e8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 -- 172.21.15.16:0/2260128575 shutdown_connections 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 -- 172.21.15.16:0/2260128575 wait complete. 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.875+0000 7f627162c740 1 -- start start 2024-04-25T01:34:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 Processor -- start 2024-04-25T01:34:14.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283ecfc0 0x55f9283f7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.855 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283f2b20 0x55f9283fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f9283f5ad0 con 0x55f9283ed9b0 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f92830cb50 con 0x55f9283f2b20 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f9283f5c50 con 0x55f9283ecfc0 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f627162c740 10 client.0 _ll_register_callbacks cb 0x55f928347ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f62677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f62677fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45622/0 (socket says 172.21.15.16:45622) 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f62677fe640 1 -- 172.21.15.16:0/3969108887 learned_addr learned my addr 172.21.15.16:0/3969108887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f6267fff640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283ecfc0 0x55f9283f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.876+0000 7f626ceb8640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283f2b20 0x55f9283fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f6267fff640 1 -- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283f2b20 msgr2=0x55f9283fe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f6267fff640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283f2b20 0x55f9283fe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.856 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f6267fff640 1 -- 172.21.15.16:0/3969108887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 msgr2=0x55f9283f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f6267fff640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f6267fff640 1 -- 172.21.15.16:0/3969108887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f928131620 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f62677fe640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f6267fff640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283ecfc0 0x55f9283f7470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62580d1430 tx=0x7f625800bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23a3611467e7f3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f62477fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f62477fe640 1 -- 172.21.15.16:0/3969108887 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625809a190 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f62477fe640 1 -- 172.21.15.16:0/3969108887 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6258096cc0 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.877+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f92835e1f0 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f62477fe640 1 -- 172.21.15.16:0/3969108887 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f625809a420 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 10 client.25429 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f9283f5ad0 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f62477fe640 1 -- 172.21.15.16:0/3969108887 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f92835e1f0 con 0x55f9283ecfc0 2024-04-25T01:34:14.857 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f62477fe640 1 -- 172.21.15.16:0/3969108887 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f62580ba2f0 con 0x55f9283ecfc0 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 10 client.25429 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 10 client.25429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 20 client.25429 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928402ee0 0x55f9284052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 4 client.25429 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.878+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f928358540 con 0x55f928402ee0 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.879+0000 7f62677fe640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928402ee0 0x55f9284052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.879+0000 7f62477fe640 10 client.25429 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.879+0000 7f62677fe640 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928402ee0 0x55f9284052d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d05b4cccf2d96de server_cookie=f27873c1ba11c8a in_seq=0 out_seq=0 2024-04-25T01:34:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.880+0000 7f62477fe640 1 -- 172.21.15.16:0/3969108887 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f928358540 con 0x55f928402ee0 2024-04-25T01:34:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.880+0000 7f62477fe640 10 client.25429 handle_command_reply: tid=0 2024-04-25T01:34:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.880+0000 7f627162c740 1 client.25429 shutdown 2024-04-25T01:34:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.880+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283ecfc0 msgr2=0x55f9283f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.880+0000 7f627162c740 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283ecfc0 0x55f9283f7470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62580d1430 tx=0x7f625800bf30 comp rx=0 tx=0).stop 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 shutdown_connections 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928402ee0 0x55f9284052d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283f2b20 0x55f9283fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ed9b0 0x55f9283f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 --2- 172.21.15.16:0/3969108887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283ecfc0 0x55f9283f7470 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 >> 172.21.15.16:0/3969108887 conn(0x55f9283e8630 msgr2=0x55f9283e7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f62477fe640 0 client.25429 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 shutdown_connections 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 1 -- 172.21.15.16:0/3969108887 wait complete. 2024-04-25T01:34:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.881+0000 7f627162c740 20 client.25429 trim_cache size 0 max 16384 2024-04-25T01:34:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.908+0000 7f627162c740 1 Processor -- start 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.908+0000 7f627162c740 1 -- start start 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f627162c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283f7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f627162c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f627162c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283eb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f627162c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62681b1710 con 0x55f9283ecfc0 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f627162c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62681aefe0 con 0x55f9283ed9b0 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f627162c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62681ace30 con 0x55f9283f2b20 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45636/0 (socket says 172.21.15.16:45636) 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 -- 172.21.15.16:0/261968883 learned_addr learned my addr 172.21.15.16:0/261968883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f626ceb8640 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283eb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 -- 172.21.15.16:0/261968883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 msgr2=0x55f9283eb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283eb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 -- 172.21.15.16:0/261968883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 msgr2=0x55f9283fe9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fe9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f6267fff640 1 -- 172.21.15.16:0/261968883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9283ee240 con 0x55f9283ecfc0 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.909+0000 7f626ceb8640 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283eb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.910+0000 7f6267fff640 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283f7790 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6258098900 tx=0x7f6258098b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929b603263260806 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.910+0000 7f624dffb640 1 -- 172.21.15.16:0/261968883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62580d1460 con 0x55f9283ecfc0 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.910+0000 7f624dffb640 1 -- 172.21.15.16:0/261968883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62580104f0 con 0x55f9283ecfc0 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.910+0000 7f624dffb640 1 -- 172.21.15.16:0/261968883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62580968f0 con 0x55f9283ecfc0 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 -- 172.21.15.16:0/261968883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 msgr2=0x55f9283f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283f7790 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6258098900 tx=0x7f6258098b60 comp rx=0 tx=0).stop 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 -- 172.21.15.16:0/261968883 shutdown_connections 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f9283eb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 --2- 172.21.15.16:0/261968883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283f7790 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 -- 172.21.15.16:0/261968883 >> 172.21.15.16:0/261968883 conn(0x55f9283e8630 msgr2=0x55f9283f7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.911+0000 7f627162c740 1 -- 172.21.15.16:0/261968883 shutdown_connections 2024-04-25T01:34:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.912+0000 7f627162c740 1 -- 172.21.15.16:0/261968883 wait complete. 2024-04-25T01:34:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.912+0000 7f627162c740 1 -- start start 2024-04-25T01:34:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.912+0000 7f627162c740 1 Processor -- start 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283fb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f928467360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f6267fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283fb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f6267fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283fb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45644/0 (socket says 172.21.15.16:45644) 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f62677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f6267fff640 1 -- 172.21.15.16:0/1477972865 learned_addr learned my addr 172.21.15.16:0/1477972865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62681ad4a0 con 0x55f9283ecfc0 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62681ae910 con 0x55f9283ed9b0 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62681adb70 con 0x55f9283f2b20 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f627162c740 10 client.0 _ll_register_callbacks cb 0x7f6260009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.913+0000 7f626ceb8640 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f928467360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f6267fff640 1 -- 172.21.15.16:0/1477972865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 msgr2=0x55f928467360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f6267fff640 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f928467360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f6267fff640 1 -- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 msgr2=0x55f9283fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f6267fff640 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f6267fff640 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9283fea70 con 0x55f9283ecfc0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f6267fff640 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283fb4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f625809a5e0 tx=0x7f62580d47c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62aefd1712199b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f62477fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f62477fe640 1 -- 172.21.15.16:0/1477972865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258099fb0 con 0x55f9283ecfc0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f62477fe640 1 -- 172.21.15.16:0/1477972865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6258083be0 con 0x55f9283ecfc0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9282d68e0 con 0x55f9283ecfc0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.914+0000 7f62477fe640 1 -- 172.21.15.16:0/1477972865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f62580968f0 con 0x55f9283ecfc0 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 10 client.15789 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f9283fea70 con 0x55f9283ecfc0 2024-04-25T01:34:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f62477fe640 1 -- 172.21.15.16:0/1477972865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6258083de0 con 0x55f9283ecfc0 2024-04-25T01:34:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f62477fe640 1 -- 172.21.15.16:0/1477972865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f625809a8b0 con 0x55f9283ecfc0 2024-04-25T01:34:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 10 client.15789 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 20 client.15789 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928468c90 0x55f92846b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 4 client.15789 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.915+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9282d68e0 con 0x55f928468c90 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.916+0000 7f62677fe640 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928468c90 0x55f92846b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.917+0000 7f62677fe640 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928468c90 0x55f92846b0b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5b37813dc55b95a server_cookie=4f112cada4d50a5a in_seq=0 out_seq=0 2024-04-25T01:34:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.917+0000 7f62477fe640 10 client.15789 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.926+0000 7f62477fe640 1 -- 172.21.15.16:0/1477972865 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9282d68e0 con 0x55f928468c90 2024-04-25T01:34:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.926+0000 7f62477fe640 10 client.15789 handle_command_reply: tid=0 2024-04-25T01:34:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.926+0000 7f627162c740 1 client.15789 shutdown 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 msgr2=0x55f9283fb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283fb4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f625809a5e0 tx=0x7f62580d47c0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 shutdown_connections 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9283f2b20 0x55f928467360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f928468c90 0x55f92846b0b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9283ed9b0 0x55f9283fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 --2- 172.21.15.16:0/1477972865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f9283ecfc0 0x55f9283fb4b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f62477fe640 0 client.15789 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 >> 172.21.15.16:0/1477972865 conn(0x55f9283e8630 msgr2=0x55f9283f7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.927+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 shutdown_connections 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.928+0000 7f627162c740 1 -- 172.21.15.16:0/1477972865 wait complete. 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.928+0000 7f627162c740 20 client.15789 trim_cache size 0 max 16384 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:34:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:34:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:34:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.929+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f62500a2c80 msgr2=0x7f62500a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.929+0000 7f626d6b9640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f62500a2c80 0x7f62500a5140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6260009a60 tx=0x7f6260010040 comp rx=0 tx=0).stop 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.929+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 msgr2=0x7f626819dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.929+0000 7f626d6b9640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 0x7f626819dcc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f625800b3c0 tx=0x7f625800b880 comp rx=0 tx=0).stop 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 shutdown_connections 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f62500a2c80 0x7f62500a5140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6268103180 0x7f62681a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62681027c0 0x7f626819e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 --2- 172.21.15.16:0/347147012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62681015d0 0x7f626819dcc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 >> 172.21.15.16:0/347147012 conn(0x7f62680fcb40 msgr2=0x7f6268103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 shutdown_connections 2024-04-25T01:34:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:14.930+0000 7f626d6b9640 1 -- 172.21.15.16:0/347147012 wait complete. 2024-04-25T01:34:15.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:14 smithi016 ceph-mon[29153]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-04-25T01:34:15.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:14 smithi038 ceph-mon[32841]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-04-25T01:34:15.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:14 smithi156 ceph-mon[32631]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 72 MiB/s wr, 1.17k op/s 2024-04-25T01:34:17.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:16 smithi016 ceph-mon[29153]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 60 MiB/s wr, 977 op/s 2024-04-25T01:34:17.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:16 smithi038 ceph-mon[32841]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 60 MiB/s wr, 977 op/s 2024-04-25T01:34:17.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:16 smithi156 ceph-mon[32631]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 60 MiB/s wr, 977 op/s 2024-04-25T01:34:19.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:18 smithi016 ceph-mon[29153]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:19.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:18 smithi038 ceph-mon[32841]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:19.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:18 smithi156 ceph-mon[32631]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-25T01:34:21.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:20 smithi016 ceph-mon[29153]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 67 MiB/s wr, 1.10k op/s 2024-04-25T01:34:21.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:20 smithi038 ceph-mon[32841]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 67 MiB/s wr, 1.10k op/s 2024-04-25T01:34:21.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:20 smithi156 ceph-mon[32631]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 67 MiB/s wr, 1.10k op/s 2024-04-25T01:34:23.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:22 smithi016 ceph-mon[29153]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 65 MiB/s wr, 1.06k op/s 2024-04-25T01:34:23.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:22 smithi038 ceph-mon[32841]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 65 MiB/s wr, 1.06k op/s 2024-04-25T01:34:23.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:22 smithi156 ceph-mon[32631]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 65 MiB/s wr, 1.06k op/s 2024-04-25T01:34:25.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:24 smithi016 ceph-mon[29153]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 70 MiB/s wr, 1.15k op/s 2024-04-25T01:34:25.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:24 smithi038 ceph-mon[32841]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 70 MiB/s wr, 1.15k op/s 2024-04-25T01:34:25.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:24 smithi156 ceph-mon[32631]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 70 MiB/s wr, 1.15k op/s 2024-04-25T01:34:27.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:26 smithi016 ceph-mon[29153]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 58 MiB/s wr, 950 op/s 2024-04-25T01:34:27.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:26 smithi038 ceph-mon[32841]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 58 MiB/s wr, 950 op/s 2024-04-25T01:34:27.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:26 smithi156 ceph-mon[32631]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 58 MiB/s wr, 950 op/s 2024-04-25T01:34:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.082+0000 7f94b159a640 1 -- 172.21.15.16:0/2858587932 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac032b40 con 0x7f94b41087d0 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.083+0000 7f94b359e640 1 -- 172.21.15.16:0/2858587932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 msgr2=0x7f94b410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.083+0000 7f94b359e640 1 --2- 172.21.15.16:0/2858587932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b410abc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f94ac005fd0 tx=0x7f94ac02b410 comp rx=0 tx=0).stop 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 -- 172.21.15.16:0/2858587932 shutdown_connections 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 --2- 172.21.15.16:0/2858587932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b410dc70 0x7f94b4110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 --2- 172.21.15.16:0/2858587932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94b410b190 0x7f94b410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 --2- 172.21.15.16:0/2858587932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b410abc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 -- 172.21.15.16:0/2858587932 >> 172.21.15.16:0/2858587932 conn(0x7f94b406e7a0 msgr2=0x7f94b4070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 -- 172.21.15.16:0/2858587932 shutdown_connections 2024-04-25T01:34:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.084+0000 7f94b359e640 1 -- 172.21.15.16:0/2858587932 wait complete. 2024-04-25T01:34:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.085+0000 7f94b359e640 1 Processor -- start 2024-04-25T01:34:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.085+0000 7f94b359e640 1 -- start start 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b41a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94b410b190 0x7f94b41a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b410dc70 0x7f94b41add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b359e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41132d0 con 0x7f94b410dc70 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b359e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4113150 con 0x7f94b410b190 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4113450 con 0x7f94b41087d0 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b41a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b41a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:57694/0 (socket says 172.21.15.16:57694) 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 -- 172.21.15.16:0/246698135 learned_addr learned my addr 172.21.15.16:0/246698135 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b410dc70 0x7f94b41add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 -- 172.21.15.16:0/246698135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94b410b190 msgr2=0x7f94b41a6d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94b410b190 0x7f94b41a6d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 -- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b410dc70 msgr2=0x7f94b41add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b410dc70 0x7f94b41add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.086+0000 7f94b259c640 1 -- 172.21.15.16:0/246698135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b41ae400 con 0x7f94b41087d0 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.087+0000 7f94b259c640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b41a67e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0326b0 tx=0x7f94ac032a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.088+0000 7f94a37fe640 1 -- 172.21.15.16:0/246698135 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac041030 con 0x7f94b41087d0 2024-04-25T01:34:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.088+0000 7f94a37fe640 1 -- 172.21.15.16:0/246698135 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94ac003e20 con 0x7f94b41087d0 2024-04-25T01:34:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.088+0000 7f94a37fe640 1 -- 172.21.15.16:0/246698135 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac037040 con 0x7f94b41087d0 2024-04-25T01:34:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.089+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b41ae690 con 0x7f94b41087d0 2024-04-25T01:34:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.089+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b41aec10 con 0x7f94b41087d0 2024-04-25T01:34:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 Processor -- start 2024-04-25T01:34:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 -- start start 2024-04-25T01:34:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 0x560202cd9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560202c35cb0 con 0x560202cd1fc0 2024-04-25T01:34:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560202bde5b0 con 0x560202cd7b20 2024-04-25T01:34:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.093+0000 7f94bba11740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560202c31c70 con 0x560202cd29b0 2024-04-25T01:34:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 0x560202cd9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 0x560202cd9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46806/0 (socket says 172.21.15.16:46806) 2024-04-25T01:34:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 -- 172.21.15.16:0/453574712 learned_addr learned my addr 172.21.15.16:0/453574712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94a37fe640 1 -- 172.21.15.16:0/246698135 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f94ac044070 con 0x7f94b41087d0 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 -- 172.21.15.16:0/453574712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 msgr2=0x560202cd74b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd74b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 -- 172.21.15.16:0/453574712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 msgr2=0x560202cd23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.094+0000 7f94b1d9b640 1 -- 172.21.15.16:0/453574712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56020294a210 con 0x560202cd7b20 2024-04-25T01:34:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.095+0000 7f94a37fe640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f949c0a2d10 0x7f949c0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.096+0000 7f94a37fe640 1 -- 172.21.15.16:0/246698135 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f94ac0ef4c0 con 0x7f94b41087d0 2024-04-25T01:34:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.096+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f949c0a2d10 0x7f949c0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.097+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f949c0a2d10 0x7f949c0a51d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x560202cd3210 tx=0x7f94a4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.098+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 0x560202cd9fb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f94a80096f0 tx=0x7f94a8009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930aeead994e2906 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.099+0000 7f94a0ff9640 1 -- 172.21.15.16:0/453574712 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8015020 con 0x560202cd7b20 2024-04-25T01:34:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.099+0000 7f94a0ff9640 1 -- 172.21.15.16:0/453574712 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94a800bd80 con 0x560202cd7b20 2024-04-25T01:34:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.099+0000 7f94a0ff9640 1 -- 172.21.15.16:0/453574712 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8015020 con 0x560202cd7b20 2024-04-25T01:34:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 -- 172.21.15.16:0/453574712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 msgr2=0x560202cd9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 0x560202cd9fb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f94a80096f0 tx=0x7f94a8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 -- 172.21.15.16:0/453574712 shutdown_connections 2024-04-25T01:34:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202cd7b20 0x560202cd9fb0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f94a80096f0 tx=0x7f94a8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 --2- 172.21.15.16:0/453574712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.101+0000 7f94bba11740 1 -- 172.21.15.16:0/453574712 >> 172.21.15.16:0/453574712 conn(0x560202ccd630 msgr2=0x560202ccda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.102+0000 7f94bba11740 1 -- 172.21.15.16:0/453574712 shutdown_connections 2024-04-25T01:34:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.103+0000 7f94bba11740 1 -- 172.21.15.16:0/453574712 wait complete. 2024-04-25T01:34:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.103+0000 7f94bba11740 1 -- start start 2024-04-25T01:34:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.103+0000 7f94bba11740 1 Processor -- start 2024-04-25T01:34:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cdc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cdcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202ce3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560202cdaad0 con 0x560202cd29b0 2024-04-25T01:34:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560202bf1b50 con 0x560202ce1bd0 2024-04-25T01:34:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560202cdac50 con 0x560202cd1fc0 2024-04-25T01:34:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94bba11740 10 client.0 _ll_register_callbacks cb 0x560202c2cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94b259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cdc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94b259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cdc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:57706/0 (socket says 172.21.15.16:57706) 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94b259c640 1 -- 172.21.15.16:0/1025112373 learned_addr learned my addr 172.21.15.16:0/1025112373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202ce3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.104+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cdcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94b259c640 1 -- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 msgr2=0x560202ce3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94b259c640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202ce3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94b259c640 1 -- 172.21.15.16:0/1025112373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 msgr2=0x560202cdcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94b259c640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cdcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94b259c640 1 -- 172.21.15.16:0/1025112373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560202a16620 con 0x560202cd1fc0 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94b259c640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cdc660 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0b8160 tx=0x7f94ac0bbb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9917c4876ac93c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f9499ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:34:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f9499ffb640 1 -- 172.21.15.16:0/1025112373 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0bbd00 con 0x560202cd1fc0 2024-04-25T01:34:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f9499ffb640 1 -- 172.21.15.16:0/1025112373 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f94ac007390 con 0x560202cd1fc0 2024-04-25T01:34:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.105+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94a800bd80 con 0x560202cd1fc0 2024-04-25T01:34:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.106+0000 7f9499ffb640 1 -- 172.21.15.16:0/1025112373 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f94ac0c00b0 con 0x560202cd1fc0 2024-04-25T01:34:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.107+0000 7f94bba11740 10 client.25444 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.107+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560202cdab90 con 0x560202cd1fc0 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.107+0000 7f9499ffb640 1 -- 172.21.15.16:0/1025112373 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94a800bd80 con 0x560202cd1fc0 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.107+0000 7f9499ffb640 1 -- 172.21.15.16:0/1025112373 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f94ac03a030 con 0x560202cd1fc0 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.108+0000 7f94bba11740 10 client.25444 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.108+0000 7f94bba11740 10 client.25444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.108+0000 7f94bba11740 20 client.25444 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.108+0000 7f94bba11740 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202ce8a70 0x560202ceaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.108+0000 7f94bba11740 4 client.25444 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.108+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560202c431f0 con 0x560202ce8a70 2024-04-25T01:34:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.109+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202ce8a70 0x560202ceaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.110+0000 7f9499ffb640 10 client.25444 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.110+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202ce8a70 0x560202ceaea0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c45733c560cfc4b server_cookie=ccc7a193d966d353 in_seq=0 out_seq=0 2024-04-25T01:34:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.111+0000 7f9499ffb640 1 -- 172.21.15.16:0/1025112373 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560202c431f0 con 0x560202ce8a70 2024-04-25T01:34:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.111+0000 7f9499ffb640 10 client.25444 handle_command_reply: tid=0 2024-04-25T01:34:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.111+0000 7f94bba11740 1 client.25444 shutdown 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 msgr2=0x560202cdc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cdc660 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0b8160 tx=0x7f94ac0bbb90 comp rx=0 tx=0).stop 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 shutdown_connections 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202ce8a70 0x560202ceaea0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202ce3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cdcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 --2- 172.21.15.16:0/1025112373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cdc660 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 >> 172.21.15.16:0/1025112373 conn(0x560202ccd630 msgr2=0x560202ccc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f9499ffb640 0 client.25444 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 shutdown_connections 2024-04-25T01:34:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.112+0000 7f94bba11740 1 -- 172.21.15.16:0/1025112373 wait complete. 2024-04-25T01:34:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.113+0000 7f94bba11740 20 client.25444 trim_cache size 0 max 16384 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.137+0000 7f94bba11740 1 Processor -- start 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 -- start start 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cdc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41ba110 con 0x560202cd1fc0 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41b79e0 con 0x560202ce1bd0 2024-04-25T01:34:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94bba11740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41b5830 con 0x560202cd29b0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:57714/0 (socket says 172.21.15.16:57714) 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94b1d9b640 1 -- 172.21.15.16:0/710230526 learned_addr learned my addr 172.21.15.16:0/710230526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cdc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.138+0000 7f94b259c640 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f94b1d9b640 1 -- 172.21.15.16:0/710230526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 msgr2=0x560202cdc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cdc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f94b1d9b640 1 -- 172.21.15.16:0/710230526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 msgr2=0x560202cd4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f94b1d9b640 1 -- 172.21.15.16:0/710230526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560202cdd420 con 0x560202cd29b0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f94b1d9b640 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd45d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f94a8006ba0 tx=0x7f94a8006b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e93ff3b026334244 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f949bfff640 1 -- 172.21.15.16:0/710230526 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8015020 con 0x560202cd29b0 2024-04-25T01:34:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f949bfff640 1 -- 172.21.15.16:0/710230526 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94a800bd80 con 0x560202cd29b0 2024-04-25T01:34:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.139+0000 7f949bfff640 1 -- 172.21.15.16:0/710230526 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8004040 con 0x560202cd29b0 2024-04-25T01:34:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 -- 172.21.15.16:0/710230526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 msgr2=0x560202cd45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd45d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f94a8006ba0 tx=0x7f94a8006b10 comp rx=0 tx=0).stop 2024-04-25T01:34:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 -- 172.21.15.16:0/710230526 shutdown_connections 2024-04-25T01:34:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cdc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd29b0 0x560202cd45d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 --2- 172.21.15.16:0/710230526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd1fc0 0x560202cd4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 -- 172.21.15.16:0/710230526 >> 172.21.15.16:0/710230526 conn(0x560202ccd630 msgr2=0x560202ceaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 -- 172.21.15.16:0/710230526 shutdown_connections 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.140+0000 7f94bba11740 1 -- 172.21.15.16:0/710230526 wait complete. 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 -- start start 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 Processor -- start 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cf9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cf9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cfa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41b5ea0 con 0x560202cd29b0 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41b7310 con 0x560202ce1bd0 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41b6570 con 0x560202cd1fc0 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94bba11740 10 client.0 _ll_register_callbacks cb 0x7f94ac005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cf9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94b259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cf9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94b259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cf9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:57720/0 (socket says 172.21.15.16:57720) 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94b259c640 1 -- 172.21.15.16:0/4074519499 learned_addr learned my addr 172.21.15.16:0/4074519499 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.141+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cfa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b2d9d640 1 -- 172.21.15.16:0/4074519499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 msgr2=0x560202cf9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cf9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b2d9d640 1 -- 172.21.15.16:0/4074519499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 msgr2=0x560202cf9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cf9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b2d9d640 1 -- 172.21.15.16:0/4074519499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560202cfaff0 con 0x560202ce1bd0 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b259c640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cf9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94b2d9d640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cfa980 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006f70 tx=0x7f94a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a9e5a9a1a5759c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f9499ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f9499ffb640 1 -- 172.21.15.16:0/4074519499 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a401c040 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94a800bd80 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.142+0000 7f9499ffb640 1 -- 172.21.15.16:0/4074519499 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f94a4024710 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f9499ffb640 1 -- 172.21.15.16:0/4074519499 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f94a4032070 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 10 client.25487 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560202cfc450 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f9499ffb640 1 -- 172.21.15.16:0/4074519499 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94a800bd80 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f9499ffb640 1 -- 172.21.15.16:0/4074519499 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f94a4014b50 con 0x560202ce1bd0 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 10 client.25487 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 10 client.25487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 20 client.25487 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202d4f0a0 0x560202d514c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 4 client.25487 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:34:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560202c3d540 con 0x560202d4f0a0 2024-04-25T01:34:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.143+0000 7f94b259c640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202d4f0a0 0x560202d514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.144+0000 7f94b259c640 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202d4f0a0 0x560202d514c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217ec7fb8efc26db server_cookie=34e559d427442cd6 in_seq=0 out_seq=0 2024-04-25T01:34:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.144+0000 7f9499ffb640 10 client.25487 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.148+0000 7f9499ffb640 1 -- 172.21.15.16:0/4074519499 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560202c3d540 con 0x560202d4f0a0 2024-04-25T01:34:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.148+0000 7f9499ffb640 10 client.25487 handle_command_reply: tid=0 2024-04-25T01:34:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.148+0000 7f94bba11740 1 client.25487 shutdown 2024-04-25T01:34:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 msgr2=0x560202cfa980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cfa980 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006f70 tx=0x7f94a40046c0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 shutdown_connections 2024-04-25T01:34:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x560202d4f0a0 0x560202d514c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560202ce1bd0 0x560202cfa980 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f9499ffb640 0 client.25487 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560202cd29b0 0x560202cf9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 --2- 172.21.15.16:0/4074519499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560202cd1fc0 0x560202cf9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.150+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 >> 172.21.15.16:0/4074519499 conn(0x560202ccd630 msgr2=0x560202cd4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.151+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 shutdown_connections 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.151+0000 7f94bba11740 1 -- 172.21.15.16:0/4074519499 wait complete. 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.151+0000 7f94bba11740 20 client.25487 trim_cache size 0 max 16384 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:34:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:34:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f949c0a2d10 msgr2=0x7f949c0a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f949c0a2d10 0x7f949c0a51d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x560202cd3210 tx=0x7f94a4008040 comp rx=0 tx=0).stop 2024-04-25T01:34:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 msgr2=0x7f94b41a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b41a67e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0326b0 tx=0x7f94ac032a90 comp rx=0 tx=0).stop 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 shutdown_connections 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f949c0a2d10 0x7f949c0a51d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b410dc70 0x7f94b41add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94b410b190 0x7f94b41a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 --2- 172.21.15.16:0/246698135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94b41087d0 0x7f94b41a67e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 >> 172.21.15.16:0/246698135 conn(0x7f94b406e7a0 msgr2=0x7f94b4109230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 shutdown_connections 2024-04-25T01:34:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:28.152+0000 7f94b359e640 1 -- 172.21.15.16:0/246698135 wait complete. 2024-04-25T01:34:29.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:28 smithi016 ceph-mon[29153]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-04-25T01:34:29.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:28 smithi038 ceph-mon[32841]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-04-25T01:34:29.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:28 smithi156 ceph-mon[32631]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-04-25T01:34:31.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:30 smithi016 ceph-mon[29153]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-04-25T01:34:31.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:30 smithi038 ceph-mon[32841]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-04-25T01:34:31.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:30 smithi156 ceph-mon[32631]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 68 MiB/s wr, 1.11k op/s 2024-04-25T01:34:33.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:32 smithi016 ceph-mon[29153]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 66 MiB/s wr, 1.08k op/s 2024-04-25T01:34:33.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:32 smithi038 ceph-mon[32841]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 66 MiB/s wr, 1.08k op/s 2024-04-25T01:34:33.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:32 smithi156 ceph-mon[32631]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 66 MiB/s wr, 1.08k op/s 2024-04-25T01:34:35.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:34 smithi016 ceph-mon[29153]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 71 MiB/s wr, 1.17k op/s 2024-04-25T01:34:35.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:34 smithi038 ceph-mon[32841]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 71 MiB/s wr, 1.17k op/s 2024-04-25T01:34:35.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:34 smithi156 ceph-mon[32631]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 71 MiB/s wr, 1.17k op/s 2024-04-25T01:34:37.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:36 smithi016 ceph-mon[29153]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 59 MiB/s wr, 980 op/s 2024-04-25T01:34:37.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:36 smithi038 ceph-mon[32841]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 59 MiB/s wr, 980 op/s 2024-04-25T01:34:37.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:36 smithi156 ceph-mon[32631]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 59 MiB/s wr, 980 op/s 2024-04-25T01:34:39.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:38 smithi016 ceph-mon[29153]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 68 MiB/s wr, 1.13k op/s 2024-04-25T01:34:39.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:38 smithi038 ceph-mon[32841]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 68 MiB/s wr, 1.13k op/s 2024-04-25T01:34:39.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:38 smithi156 ceph-mon[32631]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 68 MiB/s wr, 1.13k op/s 2024-04-25T01:34:41.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:40 smithi016 ceph-mon[29153]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 67 MiB/s wr, 1.11k op/s 2024-04-25T01:34:41.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:40 smithi038 ceph-mon[32841]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 67 MiB/s wr, 1.11k op/s 2024-04-25T01:34:41.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:40 smithi156 ceph-mon[32631]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 67 MiB/s wr, 1.11k op/s 2024-04-25T01:34:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 -- 172.21.15.16:0/2003629862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 msgr2=0x7f80f0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 --2- 172.21.15.16:0/2003629862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f0100ac0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009900 tx=0x7f80e002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 -- 172.21.15.16:0/2003629862 shutdown_connections 2024-04-25T01:34:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 --2- 172.21.15.16:0/2003629862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0102250 0x7f80f0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 --2- 172.21.15.16:0/2003629862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0101890 0x7f80f0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 --2- 172.21.15.16:0/2003629862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f0100ac0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 -- 172.21.15.16:0/2003629862 >> 172.21.15.16:0/2003629862 conn(0x7f80f00fbc10 msgr2=0x7f80f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 -- 172.21.15.16:0/2003629862 shutdown_connections 2024-04-25T01:34:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.298+0000 7f80f5309640 1 -- 172.21.15.16:0/2003629862 wait complete. 2024-04-25T01:34:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 Processor -- start 2024-04-25T01:34:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 -- start start 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0101890 0x7f80f019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102250 0x7f80f01a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010b0e0 con 0x7f80f0102250 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010af60 con 0x7f80f01006a0 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.299+0000 7f80f5309640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80f010b260 con 0x7f80f0101890 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f019e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:39682/0 (socket says 172.21.15.16:39682) 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 -- 172.21.15.16:0/2511018266 learned_addr learned my addr 172.21.15.16:0/2511018266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102250 0x7f80f01a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80ee7fc640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0101890 0x7f80f019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 -- 172.21.15.16:0/2511018266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0101890 msgr2=0x7f80f019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0101890 0x7f80f019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 -- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102250 msgr2=0x7f80f01a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102250 0x7f80f01a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 -- 172.21.15.16:0/2511018266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f01a5b70 con 0x7f80f01006a0 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80eeffd640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f019e330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f80e00314e0 tx=0x7f80e0003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.300+0000 7f80d7fff640 1 -- 172.21.15.16:0/2511018266 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e003f070 con 0x7f80f01006a0 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.301+0000 7f80d7fff640 1 -- 172.21.15.16:0/2511018266 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e0031c40 con 0x7f80f01006a0 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.301+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f01a5e00 con 0x7f80f01006a0 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.301+0000 7f80d7fff640 1 -- 172.21.15.16:0/2511018266 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0035030 con 0x7f80f01006a0 2024-04-25T01:34:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.301+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f01a6260 con 0x7f80f01006a0 2024-04-25T01:34:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80d7fff640 1 -- 172.21.15.16:0/2511018266 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f80e0042070 con 0x7f80f01006a0 2024-04-25T01:34:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 Processor -- start 2024-04-25T01:34:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 -- start start 2024-04-25T01:34:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949d4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949d6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563f94932cb0 con 0x563f949cee50 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563f948db5b0 con 0x563f949cf840 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.305+0000 7f80f8013740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563f9492ec70 con 0x563f949d49b0 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949d4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949d4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:39690/0 (socket says 172.21.15.16:39690) 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 -- 172.21.15.16:0/985699905 learned_addr learned my addr 172.21.15.16:0/985699905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80ee7fc640 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949d6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 -- 172.21.15.16:0/985699905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 msgr2=0x563f949d6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949d6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 -- 172.21.15.16:0/985699905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 msgr2=0x563f949cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 -- 172.21.15.16:0/985699905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f94647210 con 0x563f949cf840 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80d7fff640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2d30 0x7f80d80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2d30 0x7f80d80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80d7fff640 1 -- 172.21.15.16:0/2511018266 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f80e00ef790 con 0x7f80f01006a0 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.306+0000 7f80eeffd640 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949d4340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f80e00f5c20 tx=0x7f80e00b8230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eff76c9efa1e8b3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.307+0000 7f80d57fa640 1 -- 172.21.15.16:0/985699905 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0049070 con 0x563f949cf840 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.307+0000 7f80d57fa640 1 -- 172.21.15.16:0/985699905 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e00bec40 con 0x563f949cf840 2024-04-25T01:34:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.307+0000 7f80d57fa640 1 -- 172.21.15.16:0/985699905 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e00bbdf0 con 0x563f949cf840 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2d30 0x7f80d80a51f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x563f949d00a0 tx=0x7f80e8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 -- 172.21.15.16:0/985699905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 msgr2=0x563f949d4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949d4340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f80e00f5c20 tx=0x7f80e00b8230 comp rx=0 tx=0).stop 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 -- 172.21.15.16:0/985699905 shutdown_connections 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949d6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949d4340 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 --2- 172.21.15.16:0/985699905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 -- 172.21.15.16:0/985699905 >> 172.21.15.16:0/985699905 conn(0x563f949ca4c0 msgr2=0x563f949ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 -- 172.21.15.16:0/985699905 shutdown_connections 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 -- 172.21.15.16:0/985699905 wait complete. 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 -- start start 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.308+0000 7f80f8013740 1 Processor -- start 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949d9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949d9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949e05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563f949d7720 con 0x563f949cf840 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563f948eeb50 con 0x563f949d49b0 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563f949d78a0 con 0x563f949cee50 2024-04-25T01:34:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949e05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949e05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:39706/0 (socket says 172.21.15.16:39706) 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80ef7fe640 1 -- 172.21.15.16:0/1035665778 learned_addr learned my addr 172.21.15.16:0/1035665778 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.309+0000 7f80f8013740 10 client.0 _ll_register_callbacks cb 0x563f94929ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949d9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80ef7fe640 1 -- 172.21.15.16:0/1035665778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 msgr2=0x563f949d9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949d9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80ef7fe640 1 -- 172.21.15.16:0/1035665778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 msgr2=0x563f949d9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949d9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80ef7fe640 1 -- 172.21.15.16:0/1035665778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f94713620 con 0x563f949d49b0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949d9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949e05b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f80e8003f00 tx=0x7f80e8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b40d6660eacb22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1035665778 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8004b20 con 0x563f949d49b0 2024-04-25T01:34:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1035665778 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f80e8004cc0 con 0x563f949d49b0 2024-04-25T01:34:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.310+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563f949401f0 con 0x563f949d49b0 2024-04-25T01:34:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.311+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1035665778 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f80e800f870 con 0x563f949d49b0 2024-04-25T01:34:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.312+0000 7f80f8013740 10 client.25499 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.312+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563f949e3990 con 0x563f949d49b0 2024-04-25T01:34:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1035665778 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e8005020 con 0x563f949d49b0 2024-04-25T01:34:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1035665778 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f80e8027070 con 0x563f949d49b0 2024-04-25T01:34:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80f8013740 10 client.25499 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80f8013740 10 client.25499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80f8013740 20 client.25499 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80f8013740 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f949e7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80f8013740 4 client.25499 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.313+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f949401f0 con 0x563f949e50f0 2024-04-25T01:34:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.316+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f949e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.317+0000 7f80d2ff5640 10 client.25499 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.317+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f949e7560 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a306aea38f0703 server_cookie=edff2c39df683d07 in_seq=0 out_seq=0 2024-04-25T01:34:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.321+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1035665778 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f949401f0 con 0x563f949e50f0 2024-04-25T01:34:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.321+0000 7f80d2ff5640 10 client.25499 handle_command_reply: tid=0 2024-04-25T01:34:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 client.25499 shutdown 2024-04-25T01:34:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 msgr2=0x563f949e05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949e05b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f80e8003f00 tx=0x7f80e8004ab0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 shutdown_connections 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f949e7560 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80d2ff5640 0 client.25499 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949e05b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.322+0000 7f80f8013740 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949d9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.323+0000 7f80f8013740 1 --2- 172.21.15.16:0/1035665778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949d9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.323+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 >> 172.21.15.16:0/1035665778 conn(0x563f949ca4c0 msgr2=0x563f949c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.323+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 shutdown_connections 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.323+0000 7f80f8013740 1 -- 172.21.15.16:0/1035665778 wait complete. 2024-04-25T01:34:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.323+0000 7f80f8013740 20 client.25499 trim_cache size 0 max 16384 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 Processor -- start 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 -- start start 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949e0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01b1b20 con 0x563f949cee50 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01af3f0 con 0x563f949cf840 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80f8013740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01ad240 con 0x563f949d49b0 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35136/0 (socket says 172.21.15.16:35136) 2024-04-25T01:34:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.351+0000 7f80eeffd640 1 -- 172.21.15.16:0/678618864 learned_addr learned my addr 172.21.15.16:0/678618864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80ee7fc640 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949e0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80eeffd640 1 -- 172.21.15.16:0/678618864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 msgr2=0x563f949ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80eeffd640 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949ebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80eeffd640 1 -- 172.21.15.16:0/678618864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 msgr2=0x563f949e0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80eeffd640 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949e0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80eeffd640 1 -- 172.21.15.16:0/678618864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f949cfcc0 con 0x563f949cee50 2024-04-25T01:34:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.352+0000 7f80ee7fc640 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949e0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.354+0000 7f80eeffd640 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949c9440 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f80e00b8960 tx=0x7f80e002a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1cdd726fad705d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.355+0000 7f80d4ff9640 1 -- 172.21.15.16:0/678618864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0049040 con 0x563f949cee50 2024-04-25T01:34:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.355+0000 7f80d4ff9640 1 -- 172.21.15.16:0/678618864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e00f5380 con 0x563f949cee50 2024-04-25T01:34:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.355+0000 7f80d4ff9640 1 -- 172.21.15.16:0/678618864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e00dc070 con 0x563f949cee50 2024-04-25T01:34:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 -- 172.21.15.16:0/678618864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 msgr2=0x563f949c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949c9440 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f80e00b8960 tx=0x7f80e002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 -- 172.21.15.16:0/678618864 shutdown_connections 2024-04-25T01:34:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949d49b0 0x563f949ebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949cf840 0x563f949e0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 --2- 172.21.15.16:0/678618864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cee50 0x563f949c9440 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.356+0000 7f80f8013740 1 -- 172.21.15.16:0/678618864 >> 172.21.15.16:0/678618864 conn(0x563f949ca4c0 msgr2=0x563f949c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.357+0000 7f80f8013740 1 -- 172.21.15.16:0/678618864 shutdown_connections 2024-04-25T01:34:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.357+0000 7f80f8013740 1 -- 172.21.15.16:0/678618864 wait complete. 2024-04-25T01:34:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.357+0000 7f80f8013740 1 -- start start 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.358+0000 7f80f8013740 1 Processor -- start 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.358+0000 7f80f8013740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949de090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.358+0000 7f80f8013740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949de5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.358+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949de090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45282/0 (socket says 172.21.15.16:45282) 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80ee7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949de5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80ee7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949de5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35150/0 (socket says 172.21.15.16:35150) 2024-04-25T01:34:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80eeffd640 1 -- 172.21.15.16:0/1866457060 learned_addr learned my addr 172.21.15.16:0/1866457060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949df5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01ad8b0 con 0x563f949cf840 2024-04-25T01:34:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01aed20 con 0x563f949d49b0 2024-04-25T01:34:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80ef7fe640 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949df5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.359+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01adf80 con 0x563f949cee50 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80eeffd640 1 -- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 msgr2=0x563f949df5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949df5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80eeffd640 1 -- 172.21.15.16:0/1866457060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 msgr2=0x563f949de5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949de5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80eeffd640 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f949dfbd0 con 0x563f949cee50 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80f8013740 10 client.0 _ll_register_callbacks cb 0x7f80e8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80eeffd640 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949de090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80e002a9b0 tx=0x7f80e00b8360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e99434285b97de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1866457060 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e0049070 con 0x563f949cee50 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1866457060 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f80e00f5730 con 0x563f949cee50 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.360+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80e00f5380 con 0x563f949cee50 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.361+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1866457060 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f80e00d6050 con 0x563f949cee50 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.361+0000 7f80f8013740 10 client.25465 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.361+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=22}) -- 0x563f94a4b900 con 0x563f949cee50 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1866457060 <== mon.1 v2:172.21.15.156:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f80e00f5d10 con 0x563f949cee50 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80f8013740 10 client.25465 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80f8013740 10 client.25465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80f8013740 20 client.25465 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f94a4e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80f8013740 4 client.25465 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.362+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f80e00f5380 con 0x563f949e50f0 2024-04-25T01:34:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.363+0000 7f80ee7fc640 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f94a4e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.364+0000 7f80ee7fc640 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f94a4e020 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67dbe3428adc6b80 server_cookie=d440913164f9b704 in_seq=0 out_seq=0 2024-04-25T01:34:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.364+0000 7f80d2ff5640 10 client.25465 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80d2ff5640 1 -- 172.21.15.16:0/1866457060 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f80e00f5380 con 0x563f949e50f0 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80d2ff5640 10 client.25465 handle_command_reply: tid=0 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 client.25465 shutdown 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 msgr2=0x563f949de090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949de090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80e002a9b0 tx=0x7f80e00b8360 comp rx=0 tx=0).stop 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 shutdown_connections 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x563f949e50f0 0x563f94a4e020 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80d2ff5640 0 client.25465 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563f949d49b0 0x563f949df5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f949cf840 0x563f949de5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 --2- 172.21.15.16:0/1866457060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f949cee50 0x563f949de090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 >> 172.21.15.16:0/1866457060 conn(0x563f949ca4c0 msgr2=0x563f949c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 shutdown_connections 2024-04-25T01:34:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.379+0000 7f80f8013740 1 -- 172.21.15.16:0/1866457060 wait complete. 2024-04-25T01:34:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.380+0000 7f80f8013740 20 client.25465 trim_cache size 0 max 16384 2024-04-25T01:34:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:34:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:34:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:34:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.380+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2d30 msgr2=0x7f80d80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.380+0000 7f80f5309640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2d30 0x7f80d80a51f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x563f949d00a0 tx=0x7f80e8009500 comp rx=0 tx=0).stop 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 msgr2=0x7f80f019e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f019e330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f80e00314e0 tx=0x7f80e0003c10 comp rx=0 tx=0).stop 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 shutdown_connections 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f80d80a2d30 0x7f80d80a51f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f0102250 0x7f80f01a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80f0101890 0x7f80f019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 --2- 172.21.15.16:0/2511018266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f01006a0 0x7f80f019e330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.381+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 >> 172.21.15.16:0/2511018266 conn(0x7f80f00fbc10 msgr2=0x7f80f010db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.383+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 shutdown_connections 2024-04-25T01:34:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:41.383+0000 7f80f5309640 1 -- 172.21.15.16:0/2511018266 wait complete. 2024-04-25T01:34:43.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:42 smithi038 ceph-mon[32841]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 65 MiB/s wr, 1.08k op/s 2024-04-25T01:34:43.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:42 smithi156 ceph-mon[32631]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 65 MiB/s wr, 1.08k op/s 2024-04-25T01:34:43.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:42 smithi016 ceph-mon[29153]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 65 MiB/s wr, 1.08k op/s 2024-04-25T01:34:45.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:44 smithi038 ceph-mon[32841]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 70 MiB/s wr, 1.17k op/s 2024-04-25T01:34:45.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:44 smithi156 ceph-mon[32631]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 70 MiB/s wr, 1.17k op/s 2024-04-25T01:34:45.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:44 smithi016 ceph-mon[29153]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 70 MiB/s wr, 1.17k op/s 2024-04-25T01:34:47.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:46 smithi038 ceph-mon[32841]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 58 MiB/s wr, 962 op/s 2024-04-25T01:34:47.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:46 smithi156 ceph-mon[32631]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 58 MiB/s wr, 962 op/s 2024-04-25T01:34:47.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:46 smithi016 ceph-mon[29153]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 58 MiB/s wr, 962 op/s 2024-04-25T01:34:49.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:48 smithi038 ceph-mon[32841]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-04-25T01:34:49.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:48 smithi156 ceph-mon[32631]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-04-25T01:34:49.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:48 smithi016 ceph-mon[29153]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-04-25T01:34:50.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:49 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:50.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:49 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:50.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:49 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:50 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-04-25T01:34:51.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:50 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 67 MiB/s wr, 1.12k op/s 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:51.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:50 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: Unable to set osd_memory_target on smithi156 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:52.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:51 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:34:52.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-25T01:34:52.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T01:34:52.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T01:34:52.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: Unable to set osd_memory_target on smithi156 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T01:34:52.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:52.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:52.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: Unable to set osd_memory_target on smithi156 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:34:53.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:53 smithi016 ceph-mon[29153]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 64 MiB/s wr, 1.08k op/s 2024-04-25T01:34:53.463 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:53 smithi038 ceph-mon[32841]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 64 MiB/s wr, 1.08k op/s 2024-04-25T01:34:53.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:53 smithi156 ceph-mon[32631]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 64 MiB/s wr, 1.08k op/s 2024-04-25T01:34:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.541+0000 7fea99851640 1 -- 172.21.15.16:0/1789788939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a4c40 msgr2=0x7fea8c0a5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.541+0000 7fea99851640 1 --2- 172.21.15.16:0/1789788939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a4c40 0x7fea8c0a5060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea88009900 tx=0x7fea8802a400 comp rx=0 tx=0).stop 2024-04-25T01:34:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.542+0000 7fea99851640 1 -- 172.21.15.16:0/1789788939 shutdown_connections 2024-04-25T01:34:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.542+0000 7fea99851640 1 --2- 172.21.15.16:0/1789788939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a6710 0x7fea8c0aafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.542+0000 7fea99851640 1 --2- 172.21.15.16:0/1789788939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.542+0000 7fea99851640 1 --2- 172.21.15.16:0/1789788939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a4c40 0x7fea8c0a5060 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.542+0000 7fea99851640 1 -- 172.21.15.16:0/1789788939 >> 172.21.15.16:0/1789788939 conn(0x7fea8c0a00d0 msgr2=0x7fea8c0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.543+0000 7fea99851640 1 -- 172.21.15.16:0/1789788939 shutdown_connections 2024-04-25T01:34:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.544+0000 7fea99851640 1 -- 172.21.15.16:0/1789788939 wait complete. 2024-04-25T01:34:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 Processor -- start 2024-04-25T01:34:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 -- start start 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8c0a4c40 0x7fea8c0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a6710 0x7fea8c1535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0b8ef0 con 0x7fea8c0a4c40 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0b8d70 con 0x7fea8c0a6710 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99851640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0b9070 con 0x7fea8c0a5d70 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea93fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea93fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45900/0 (socket says 172.21.15.16:45900) 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea93fff640 1 -- 172.21.15.16:0/720968599 learned_addr learned my addr 172.21.15.16:0/720968599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea99050640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a6710 0x7fea8c1535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.545+0000 7fea9884f640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8c0a4c40 0x7fea8c0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea93fff640 1 -- 172.21.15.16:0/720968599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a6710 msgr2=0x7fea8c1535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea93fff640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a6710 0x7fea8c1535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea93fff640 1 -- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8c0a4c40 msgr2=0x7fea8c0b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea93fff640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8c0a4c40 0x7fea8c0b1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea93fff640 1 -- 172.21.15.16:0/720968599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c153b20 con 0x7fea8c0a5d70 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea99050640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a6710 0x7fea8c1535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea93fff640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0b1d80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea8000e9f0 tx=0x7fea8000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea91ffb640 1 -- 172.21.15.16:0/720968599 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8000cd80 con 0x7fea8c0a5d70 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8c153cf0 con 0x7fea8c0a5d70 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea91ffb640 1 -- 172.21.15.16:0/720968599 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea800108e0 con 0x7fea8c0a5d70 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.546+0000 7fea91ffb640 1 -- 172.21.15.16:0/720968599 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80018920 con 0x7fea8c0a5d70 2024-04-25T01:34:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.547+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8c154200 con 0x7fea8c0a5d70 2024-04-25T01:34:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.549+0000 7fea91ffb640 1 -- 172.21.15.16:0/720968599 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fea80010a90 con 0x7fea8c0a5d70 2024-04-25T01:34:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.550+0000 7fea91ffb640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea840a2c60 0x7fea840a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.550+0000 7fea91ffb640 1 -- 172.21.15.16:0/720968599 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea800cf290 con 0x7fea8c0a5d70 2024-04-25T01:34:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.550+0000 7fea9884f640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea840a2c60 0x7fea840a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.550+0000 7fea9d7bf740 1 Processor -- start 2024-04-25T01:34:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.550+0000 7fea9d7bf740 1 -- start start 2024-04-25T01:34:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a5f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a644b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42a66fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557f429c2cb0 con 0x557f42a5efc0 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557f4296b5b0 con 0x557f42a5f9b0 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f429bec70 con 0x557f42a64b20 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42a66fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42a66fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45906/0 (socket says 172.21.15.16:45906) 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 -- 172.21.15.16:0/2033115413 learned_addr learned my addr 172.21.15.16:0/2033115413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea93fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a644b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9884f640 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a5f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea9884f640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea840a2c60 0x7fea840a5120 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea88005ab0 tx=0x7fea88000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 -- 172.21.15.16:0/2033115413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 msgr2=0x557f42a644b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a644b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 -- 172.21.15.16:0/2033115413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 msgr2=0x557f42a5f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a5f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea99050640 1 -- 172.21.15.16:0/2033115413 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f426d7210 con 0x557f42a64b20 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.551+0000 7fea93fff640 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a644b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.552+0000 7fea99050640 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42a66fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea94067450 tx=0x7fea9404faf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e2a5a37c47f5bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.552+0000 7fea7effd640 1 -- 172.21.15.16:0/2033115413 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94081070 con 0x557f42a64b20 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.552+0000 7fea7effd640 1 -- 172.21.15.16:0/2033115413 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea94070da0 con 0x557f42a64b20 2024-04-25T01:34:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.552+0000 7fea7effd640 1 -- 172.21.15.16:0/2033115413 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9406a070 con 0x557f42a64b20 2024-04-25T01:34:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2033115413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 msgr2=0x557f42a66fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42a66fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea94067450 tx=0x7fea9404faf0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2033115413 shutdown_connections 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42a66fb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a644b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2033115413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a5f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2033115413 >> 172.21.15.16:0/2033115413 conn(0x557f42a5a630 msgr2=0x557f42a5aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2033115413 shutdown_connections 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.553+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2033115413 wait complete. 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.554+0000 7fea9d7bf740 1 -- start start 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.554+0000 7fea9d7bf740 1 Processor -- start 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a69930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a70970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557f42a67ad0 con 0x557f42a5f9b0 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557f4297eb50 con 0x557f42a64b20 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f42a67c50 con 0x557f42a5efc0 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea9d7bf740 10 client.0 _ll_register_callbacks cb 0x557f429b9ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a69930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a69930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54184/0 (socket says 172.21.15.16:54184) 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea93fff640 1 -- 172.21.15.16:0/2980480757 learned_addr learned my addr 172.21.15.16:0/2980480757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.555+0000 7fea99050640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a70970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea93fff640 1 -- 172.21.15.16:0/2980480757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 msgr2=0x557f42a693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea9884f640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea93fff640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea93fff640 1 -- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 msgr2=0x557f42a70970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea93fff640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a70970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea93fff640 1 -- 172.21.15.16:0/2980480757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f427a3620 con 0x557f42a5f9b0 2024-04-25T01:34:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.556+0000 7fea99050640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a70970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.557+0000 7fea93fff640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a69930 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fea8009b880 tx=0x7fea8009bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523aec7a6c0c65db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.557+0000 7fea73fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.557+0000 7fea73fff640 1 -- 172.21.15.16:0/2980480757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80010430 con 0x557f42a5f9b0 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.557+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea94070da0 con 0x557f42a5f9b0 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.557+0000 7fea73fff640 1 -- 172.21.15.16:0/2980480757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea800105d0 con 0x557f42a5f9b0 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.558+0000 7fea73fff640 1 -- 172.21.15.16:0/2980480757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fea800ba2b0 con 0x557f42a5f9b0 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.558+0000 7fea9d7bf740 10 client.15831 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.558+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557f42a73d50 con 0x557f42a5f9b0 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.558+0000 7fea73fff640 1 -- 172.21.15.16:0/2980480757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea94070da0 con 0x557f42a5f9b0 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.558+0000 7fea73fff640 1 -- 172.21.15.16:0/2980480757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fea80018ac0 con 0x557f42a5f9b0 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9d7bf740 10 client.15831 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9d7bf740 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9d7bf740 20 client.15831 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42a754e0 0x557f42a77980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9d7bf740 4 client.15831 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f429d01f0 con 0x557f42a754e0 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.559+0000 7fea9884f640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42a754e0 0x557f42a77980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.560+0000 7fea9884f640 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42a754e0 0x557f42a77980 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d404abadaf70e4 server_cookie=6aa23507d407e04a in_seq=0 out_seq=0 2024-04-25T01:34:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.560+0000 7fea73fff640 10 client.15831 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea73fff640 1 -- 172.21.15.16:0/2980480757 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f429d01f0 con 0x557f42a754e0 2024-04-25T01:34:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea73fff640 10 client.15831 handle_command_reply: tid=0 2024-04-25T01:34:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 client.15831 shutdown 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 msgr2=0x557f42a69930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a69930 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fea8009b880 tx=0x7fea8009bc40 comp rx=0 tx=0).stop 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 shutdown_connections 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42a754e0 0x557f42a77980 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a70970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a69930 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/2980480757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 >> 172.21.15.16:0/2980480757 conn(0x557f42a5a630 msgr2=0x557f42a59830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea73fff640 0 client.15831 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 shutdown_connections 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.561+0000 7fea9d7bf740 1 -- 172.21.15.16:0/2980480757 wait complete. 2024-04-25T01:34:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.562+0000 7fea9d7bf740 20 client.15831 trim_cache size 0 max 16384 2024-04-25T01:34:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.591+0000 7fea9d7bf740 1 Processor -- start 2024-04-25T01:34:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 -- start start 2024-04-25T01:34:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a60db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a70930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c15f9d0 con 0x557f42a5f9b0 2024-04-25T01:34:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c15d2a0 con 0x557f42a64b20 2024-04-25T01:34:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9d7bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c15b0f0 con 0x557f42a5efc0 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea99050640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9884f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a60db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea99050640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46800/0 (socket says 172.21.15.16:46800) 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea9884f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a60db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45932/0 (socket says 172.21.15.16:45932) 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea99050640 1 -- 172.21.15.16:0/3735222388 learned_addr learned my addr 172.21.15.16:0/3735222388 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.592+0000 7fea93fff640 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a70930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea93fff640 1 -- 172.21.15.16:0/3735222388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 msgr2=0x557f42a60db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea93fff640 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a60db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea93fff640 1 -- 172.21.15.16:0/3735222388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 msgr2=0x557f42a612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea93fff640 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea93fff640 1 -- 172.21.15.16:0/3735222388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f42a5a170 con 0x557f42a5f9b0 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea9884f640 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a60db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea93fff640 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a70930 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fea8000e9c0 tx=0x7fea800a48c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a657114a23069534 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.593+0000 7fea7e7fc640 1 -- 172.21.15.16:0/3735222388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea800ba450 con 0x557f42a5f9b0 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.594+0000 7fea7e7fc640 1 -- 172.21.15.16:0/3735222388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea8009aa40 con 0x557f42a5f9b0 2024-04-25T01:34:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.594+0000 7fea7e7fc640 1 -- 172.21.15.16:0/3735222388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8009b230 con 0x557f42a5f9b0 2024-04-25T01:34:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.595+0000 7fea9d7bf740 1 -- 172.21.15.16:0/3735222388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 msgr2=0x557f42a70930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.595+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a70930 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fea8000e9c0 tx=0x7fea800a48c0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.596+0000 7fea9d7bf740 1 -- 172.21.15.16:0/3735222388 shutdown_connections 2024-04-25T01:34:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.597+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a64b20 0x557f42a612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.597+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5f9b0 0x557f42a70930 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.597+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/3735222388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a5efc0 0x557f42a60db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.597+0000 7fea9d7bf740 1 -- 172.21.15.16:0/3735222388 >> 172.21.15.16:0/3735222388 conn(0x557f42a5a630 msgr2=0x557f42a693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.597+0000 7fea9d7bf740 1 -- 172.21.15.16:0/3735222388 shutdown_connections 2024-04-25T01:34:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.597+0000 7fea9d7bf740 1 -- 172.21.15.16:0/3735222388 wait complete. 2024-04-25T01:34:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.598+0000 7fea9d7bf740 1 -- start start 2024-04-25T01:34:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.598+0000 7fea9d7bf740 1 Processor -- start 2024-04-25T01:34:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a6d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a6da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea9d7bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42ad9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea93fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a6da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea93fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a6da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:46802/0 (socket says 172.21.15.16:46802) 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea93fff640 1 -- 172.21.15.16:0/952689821 learned_addr learned my addr 172.21.15.16:0/952689821 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea99050640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42ad9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c15b760 con 0x557f42a5efc0 2024-04-25T01:34:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.599+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c15cbd0 con 0x557f42a5f9b0 2024-04-25T01:34:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.600+0000 7fea9884f640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a6d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.600+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c15be30 con 0x557f42a64b20 2024-04-25T01:34:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.600+0000 7fea9d7bf740 10 client.0 _ll_register_callbacks cb 0x7fea8009b9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.601+0000 7fea99050640 1 -- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 msgr2=0x557f42a6da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.601+0000 7fea99050640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a6da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.601+0000 7fea99050640 1 -- 172.21.15.16:0/952689821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 msgr2=0x557f42a6d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.601+0000 7fea99050640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a6d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.601+0000 7fea99050640 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f42a70b50 con 0x557f42a64b20 2024-04-25T01:34:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.602+0000 7fea99050640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42ad9af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea94050770 tx=0x7fea940676b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e390b0915230f879 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.602+0000 7fea73fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.602+0000 7fea73fff640 1 -- 172.21.15.16:0/952689821 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9406a050 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.602+0000 7fea73fff640 1 -- 172.21.15.16:0/952689821 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea9407cb80 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.602+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea8009aa40 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea73fff640 1 -- 172.21.15.16:0/952689821 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fea94081040 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 10 client.25480 fetch_fsmap learned FSMap version 22 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557f42adb020 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea73fff640 1 -- 172.21.15.16:0/952689821 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea8009aa40 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea73fff640 1 -- 172.21.15.16:0/952689821 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fea94087420 con 0x557f42a64b20 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 10 client.25480 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 10 client.25480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 20 client.25480 populate_metadata read hostname 'smithi016' 2024-04-25T01:34:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42adb690 0x557f42addab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 4 client.25480 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:34:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.603+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f429ca540 con 0x557f42adb690 2024-04-25T01:34:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.604+0000 7fea9884f640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42adb690 0x557f42addab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.604+0000 7fea9884f640 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42adb690 0x557f42addab0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a5f0fdba4993ad5 server_cookie=d9da5921b4bd99c8 in_seq=0 out_seq=0 2024-04-25T01:34:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.604+0000 7fea73fff640 10 client.25480 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.616+0000 7fea73fff640 1 -- 172.21.15.16:0/952689821 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f429ca540 con 0x557f42adb690 2024-04-25T01:34:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.616+0000 7fea73fff640 10 client.25480 handle_command_reply: tid=0 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.616+0000 7fea9d7bf740 1 client.25480 shutdown 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 msgr2=0x557f42ad9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42ad9af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea94050770 tx=0x7fea940676b0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 shutdown_connections 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f42a64b20 0x557f42ad9af0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x557f42adb690 0x557f42addab0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f42a5f9b0 0x557f42a6da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 --2- 172.21.15.16:0/952689821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557f42a5efc0 0x557f42a6d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 >> 172.21.15.16:0/952689821 conn(0x557f42a5a630 msgr2=0x557f42a60db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.617+0000 7fea73fff640 0 client.25480 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:34:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.618+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 shutdown_connections 2024-04-25T01:34:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.618+0000 7fea9d7bf740 1 -- 172.21.15.16:0/952689821 wait complete. 2024-04-25T01:34:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.618+0000 7fea9d7bf740 20 client.25480 trim_cache size 0 max 16384 2024-04-25T01:34:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:34:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:34:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:34:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.619+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea840a2c60 msgr2=0x7fea840a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.619+0000 7fea99851640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea840a2c60 0x7fea840a5120 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea88005ab0 tx=0x7fea88000bc0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.619+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 msgr2=0x7fea8c0b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.619+0000 7fea99851640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0b1d80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea8000e9f0 tx=0x7fea8000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 shutdown_connections 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fea840a2c60 0x7fea840a5120 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea8c0a6710 0x7fea8c1535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea8c0a5d70 0x7fea8c0b1d80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 --2- 172.21.15.16:0/720968599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea8c0a4c40 0x7fea8c0b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 >> 172.21.15.16:0/720968599 conn(0x7fea8c0a00d0 msgr2=0x7fea8c0a6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 shutdown_connections 2024-04-25T01:34:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:34:54.620+0000 7fea99851640 1 -- 172.21.15.16:0/720968599 wait complete. 2024-04-25T01:34:55.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:55 smithi016 ceph-mon[29153]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 69 MiB/s wr, 1.16k op/s 2024-04-25T01:34:55.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:55 smithi038 ceph-mon[32841]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 69 MiB/s wr, 1.16k op/s 2024-04-25T01:34:55.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:55 smithi156 ceph-mon[32631]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 69 MiB/s wr, 1.16k op/s 2024-04-25T01:34:57.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:57 smithi016 ceph-mon[29153]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 57 MiB/s wr, 962 op/s 2024-04-25T01:34:57.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:57 smithi038 ceph-mon[32841]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 57 MiB/s wr, 962 op/s 2024-04-25T01:34:57.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:57 smithi156 ceph-mon[32631]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 57 MiB/s wr, 962 op/s 2024-04-25T01:34:59.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:34:59 smithi016 ceph-mon[29153]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 66 MiB/s wr, 1.12k op/s 2024-04-25T01:34:59.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:34:59 smithi038 ceph-mon[32841]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 66 MiB/s wr, 1.12k op/s 2024-04-25T01:34:59.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:34:59 smithi156 ceph-mon[32631]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 66 MiB/s wr, 1.12k op/s 2024-04-25T01:35:01.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:01 smithi016 ceph-mon[29153]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:01.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:01 smithi038 ceph-mon[32841]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:01.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:01 smithi156 ceph-mon[32631]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:03.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:03 smithi016 ceph-mon[29153]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 63 MiB/s wr, 1.07k op/s 2024-04-25T01:35:03.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:03 smithi038 ceph-mon[32841]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 63 MiB/s wr, 1.07k op/s 2024-04-25T01:35:03.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:03 smithi156 ceph-mon[32631]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 63 MiB/s wr, 1.07k op/s 2024-04-25T01:35:05.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:05 smithi016 ceph-mon[29153]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:05.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:05 smithi038 ceph-mon[32841]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:05.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:05 smithi156 ceph-mon[32631]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:07.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:07 smithi016 ceph-mon[29153]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 57 MiB/s wr, 959 op/s 2024-04-25T01:35:07.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:07 smithi038 ceph-mon[32841]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 57 MiB/s wr, 959 op/s 2024-04-25T01:35:07.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:07 smithi156 ceph-mon[32631]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 57 MiB/s wr, 959 op/s 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 -- 172.21.15.16:0/1033319938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e601090a0 msgr2=0x7f7e6010fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/1033319938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e601090a0 0x7f7e6010fa80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009860 tx=0x7f7e4802a360 comp rx=0 tx=0).stop 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 -- 172.21.15.16:0/1033319938 shutdown_connections 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/1033319938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e6010ffc0 0x7f7e601123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/1033319938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e601090a0 0x7f7e6010fa80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/1033319938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e60108740 0x7f7e60108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.786+0000 7f7e64ff4640 1 -- 172.21.15.16:0/1033319938 >> 172.21.15.16:0/1033319938 conn(0x7f7e6006e7b0 msgr2=0x7f7e60070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.787+0000 7f7e64ff4640 1 -- 172.21.15.16:0/1033319938 shutdown_connections 2024-04-25T01:35:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.787+0000 7f7e64ff4640 1 -- 172.21.15.16:0/1033319938 wait complete. 2024-04-25T01:35:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.788+0000 7f7e64ff4640 1 Processor -- start 2024-04-25T01:35:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.788+0000 7f7e64ff4640 1 -- start start 2024-04-25T01:35:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.788+0000 7f7e64ff4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e60108740 0x7f7e601a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.788+0000 7f7e64ff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 0x7f7e601a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.788+0000 7f7e64ff4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e6010ffc0 0x7f7e601adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e64ff4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6010b380 con 0x7f7e601090a0 2024-04-25T01:35:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e64ff4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6010b200 con 0x7f7e6010ffc0 2024-04-25T01:35:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e64ff4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6010b500 con 0x7f7e60108740 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 0x7f7e601a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 0x7f7e601a6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32884/0 (socket says 172.21.15.16:32884) 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e55d9b640 1 -- 172.21.15.16:0/640823269 learned_addr learned my addr 172.21.15.16:0/640823269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e6010ffc0 0x7f7e601adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e60108740 0x7f7e601a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e55d9b640 1 -- 172.21.15.16:0/640823269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e60108740 msgr2=0x7f7e601a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.789+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e60108740 0x7f7e601a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e55d9b640 1 -- 172.21.15.16:0/640823269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e6010ffc0 msgr2=0x7f7e601adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e6010ffc0 0x7f7e601adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e55d9b640 1 -- 172.21.15.16:0/640823269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e601ae310 con 0x7f7e601090a0 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e6010ffc0 0x7f7e601adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 0x7f7e601a6c30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009980 tx=0x7f7e48032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e57fff640 1 -- 172.21.15.16:0/640823269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48037040 con 0x7f7e601090a0 2024-04-25T01:35:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e57fff640 1 -- 172.21.15.16:0/640823269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e48041070 con 0x7f7e601090a0 2024-04-25T01:35:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e57fff640 1 -- 172.21.15.16:0/640823269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48005340 con 0x7f7e601090a0 2024-04-25T01:35:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e601ae5a0 con 0x7f7e601090a0 2024-04-25T01:35:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.790+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e601ae9e0 con 0x7f7e601090a0 2024-04-25T01:35:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.793+0000 7f7e67dd6740 1 Processor -- start 2024-04-25T01:35:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 -- start start 2024-04-25T01:35:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e17270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e57fff640 1 -- 172.21.15.16:0/640823269 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7e480040d0 con 0x7f7e601090a0 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e1c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e1ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559006d7acb0 con 0x559006e1c9b0 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559006d235b0 con 0x559006e17840 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559006d76c70 con 0x559006e16e50 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e5ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e17270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e5ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e17270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:56592/0 (socket says 172.21.15.16:56592) 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/2814937046 learned_addr learned my addr 172.21.15.16:0/2814937046 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e1ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.794+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e1c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/2814937046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 msgr2=0x559006e1c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e1c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/2814937046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 msgr2=0x559006e1ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e1ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/2814937046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559006a8f210 con 0x559006e16e50 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e17270 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7e580117e0 tx=0x7f7e58011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f502c9793ff0db6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e57fff640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7e4c0a2d00 0x7f7e4c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e54d99640 1 -- 172.21.15.16:0/2814937046 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58011310 con 0x559006e16e50 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e54d99640 1 -- 172.21.15.16:0/2814937046 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e58003e20 con 0x559006e16e50 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e54d99640 1 -- 172.21.15.16:0/2814937046 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580044f0 con 0x559006e16e50 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e57fff640 1 -- 172.21.15.16:0/640823269 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7e480eedd0 con 0x7f7e601090a0 2024-04-25T01:35:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.795+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7e4c0a2d00 0x7f7e4c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7e4c0a2d00 0x7f7e4c0a51c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x559006e17c40 tx=0x7f7e5000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 -- 172.21.15.16:0/2814937046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 msgr2=0x559006e17270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e17270 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7e580117e0 tx=0x7f7e58011ca0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 -- 172.21.15.16:0/2814937046 shutdown_connections 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e1ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e1c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/2814937046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e17270 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 -- 172.21.15.16:0/2814937046 >> 172.21.15.16:0/2814937046 conn(0x559006e124c0 msgr2=0x559006e128c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 -- 172.21.15.16:0/2814937046 shutdown_connections 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.796+0000 7f7e67dd6740 1 -- 172.21.15.16:0/2814937046 wait complete. 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 -- start start 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 Processor -- start 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e21450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e21990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e289d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559006e1fa00 con 0x559006e1c9b0 2024-04-25T01:35:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e5ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e5e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e21450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e5e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e21450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:56606/0 (socket says 172.21.15.16:56606) 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e5ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e289d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32904/0 (socket says 172.21.15.16:32904) 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e5e59c640 1 -- 172.21.15.16:0/670718882 learned_addr learned my addr 172.21.15.16:0/670718882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e21990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559006d36b50 con 0x559006e17840 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559006e1fb80 con 0x559006e16e50 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.797+0000 7f7e67dd6740 10 client.0 _ll_register_callbacks cb 0x559006d71ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e5e59c640 1 -- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 msgr2=0x559006e21990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e21990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e5e59c640 1 -- 172.21.15.16:0/670718882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 msgr2=0x559006e289d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e289d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e5e59c640 1 -- 172.21.15.16:0/670718882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559006b5b620 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e21990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e21450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7e50004650 tx=0x7f7e50004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=642d8d014453f44a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e45ffb640 1 -- 172.21.15.16:0/670718882 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5002c060 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.798+0000 7f7e45ffb640 1 -- 172.21.15.16:0/670718882 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7e50017030 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e58003e20 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e45ffb640 1 -- 172.21.15.16:0/670718882 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7e5001fc90 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 10 client.25492 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x559006e2be60 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e45ffb640 1 -- 172.21.15.16:0/670718882 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e5001fe30 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e45ffb640 1 -- 172.21.15.16:0/670718882 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7e50011600 con 0x559006e16e50 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 10 client.25492 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 10 client.25492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 20 client.25492 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e2f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 4 client.25492 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.799+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7e58003e20 con 0x559006e2d4e0 2024-04-25T01:35:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.800+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e2f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.800+0000 7f7e45ffb640 10 client.25492 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.800+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e2f930 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b2633c305886eed server_cookie=9d79254303187b70 in_seq=0 out_seq=0 2024-04-25T01:35:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e45ffb640 1 -- 172.21.15.16:0/670718882 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f7e58003e20 con 0x559006e2d4e0 2024-04-25T01:35:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e45ffb640 10 client.25492 handle_command_reply: tid=0 2024-04-25T01:35:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e67dd6740 1 client.25492 shutdown 2024-04-25T01:35:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 msgr2=0x559006e21450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e21450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7e50004650 tx=0x7f7e50004730 comp rx=0 tx=0).stop 2024-04-25T01:35:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 shutdown_connections 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e289d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.801+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e2f930 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e45ffb640 0 client.25492 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e21990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/670718882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e21450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 >> 172.21.15.16:0/670718882 conn(0x559006e124c0 msgr2=0x559006e115c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 shutdown_connections 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e67dd6740 1 -- 172.21.15.16:0/670718882 wait complete. 2024-04-25T01:35:07.783 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.802+0000 7f7e67dd6740 20 client.25492 trim_cache size 0 max 16384 2024-04-25T01:35:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.836+0000 7f7e67dd6740 1 Processor -- start 2024-04-25T01:35:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.836+0000 7f7e67dd6740 1 -- start start 2024-04-25T01:35:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e218f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 0x559006e28990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e1c9b0 0x559006e1ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601ba330 con 0x559006e17840 2024-04-25T01:35:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 0x559006e28990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e55d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 0x559006e28990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32914/0 (socket says 172.21.15.16:32914) 2024-04-25T01:35:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e55d9b640 1 -- 172.21.15.16:0/3488037289 learned_addr learned my addr 172.21.15.16:0/3488037289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e1c9b0 0x559006e1ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601b7c00 con 0x559006e1c9b0 2024-04-25T01:35:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.837+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601b5a50 con 0x559006e16e50 2024-04-25T01:35:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/3488037289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 msgr2=0x559006e218f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e218f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/3488037289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 msgr2=0x559006e28990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 0x559006e28990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/3488037289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559006e13230 con 0x559006e1c9b0 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 0x559006e28990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.838+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e1c9b0 0x559006e1ff50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e58000f80 tx=0x7f7e58005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f566c6170716d581 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.839+0000 7f7e47fff640 1 -- 172.21.15.16:0/3488037289 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58003970 con 0x559006e1c9b0 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.839+0000 7f7e47fff640 1 -- 172.21.15.16:0/3488037289 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e5800f9a0 con 0x559006e1c9b0 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.839+0000 7f7e47fff640 1 -- 172.21.15.16:0/3488037289 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580055f0 con 0x559006e1c9b0 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e1c9b0 msgr2=0x559006e1ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e1c9b0 0x559006e1ff50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e58000f80 tx=0x7f7e58005bd0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 shutdown_connections 2024-04-25T01:35:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e1c9b0 0x559006e1ff50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e17840 0x559006e28990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/3488037289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 >> 172.21.15.16:0/3488037289 conn(0x559006e124c0 msgr2=0x559006e128a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 shutdown_connections 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 -- 172.21.15.16:0/3488037289 wait complete. 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.840+0000 7f7e67dd6740 1 -- start start 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 Processor -- start 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e25ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e26000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e91af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601b60c0 con 0x559006e1c9b0 2024-04-25T01:35:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601b7530 con 0x559006e17840 2024-04-25T01:35:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601b6790 con 0x559006e16e50 2024-04-25T01:35:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e67dd6740 10 client.0 _ll_register_callbacks cb 0x7f7e50005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e5ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e91af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e5ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e91af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32930/0 (socket says 172.21.15.16:32930) 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/987098476 learned_addr learned my addr 172.21.15.16:0/987098476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.841+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e25ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e55d9b640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e26000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/987098476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 msgr2=0x559006e25ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e25ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 msgr2=0x559006e26000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e26000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e5ed9d640 1 -- 172.21.15.16:0/987098476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559006e920b0 con 0x559006e1c9b0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e5ed9d640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e91af0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7e58011620 tx=0x7f7e58004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7565aadaf5d334f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.842+0000 7f7e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.843+0000 7f7e45ffb640 1 -- 172.21.15.16:0/987098476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5800fe60 con 0x559006e1c9b0 2024-04-25T01:35:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.843+0000 7f7e45ffb640 1 -- 172.21.15.16:0/987098476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7e58011420 con 0x559006e1c9b0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.843+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e5800f9a0 con 0x559006e1c9b0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.843+0000 7f7e45ffb640 1 -- 172.21.15.16:0/987098476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7e58005580 con 0x559006e1c9b0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.843+0000 7f7e67dd6740 10 client.15855 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.843+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x559006e933c0 con 0x559006e1c9b0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e45ffb640 1 -- 172.21.15.16:0/987098476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e58033070 con 0x559006e1c9b0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e45ffb640 1 -- 172.21.15.16:0/987098476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7e58018020 con 0x559006e1c9b0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e67dd6740 10 client.15855 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e67dd6740 10 client.15855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e67dd6740 20 client.15855 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e95ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.844+0000 7f7e67dd6740 4 client.15855 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.845+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7e5800f9a0 con 0x559006e2d4e0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.845+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e95ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.845+0000 7f7e5e59c640 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e95ae0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=602efa9d57d47349 server_cookie=6d1e4a8e0ae9ef38 in_seq=0 out_seq=0 2024-04-25T01:35:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.845+0000 7f7e45ffb640 10 client.15855 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.849+0000 7f7e45ffb640 1 -- 172.21.15.16:0/987098476 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7e5800f9a0 con 0x559006e2d4e0 2024-04-25T01:35:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.849+0000 7f7e45ffb640 10 client.15855 handle_command_reply: tid=0 2024-04-25T01:35:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.849+0000 7f7e67dd6740 1 client.15855 shutdown 2024-04-25T01:35:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 msgr2=0x559006e91af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e91af0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7e58011620 tx=0x7f7e58004e70 comp rx=0 tx=0).stop 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 shutdown_connections 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559006e1c9b0 0x559006e91af0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x559006e2d4e0 0x559006e95ae0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559006e17840 0x559006e26000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e45ffb640 0 client.15855 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 --2- 172.21.15.16:0/987098476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559006e16e50 0x559006e25ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 >> 172.21.15.16:0/987098476 conn(0x559006e124c0 msgr2=0x559006e10d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.850+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 shutdown_connections 2024-04-25T01:35:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e67dd6740 1 -- 172.21.15.16:0/987098476 wait complete. 2024-04-25T01:35:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e67dd6740 20 client.15855 trim_cache size 0 max 16384 2024-04-25T01:35:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:35:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:35:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:35:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7e4c0a2d00 msgr2=0x7f7e4c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7e4c0a2d00 0x7f7e4c0a51c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x559006e17c40 tx=0x7f7e5000b390 comp rx=0 tx=0).stop 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 msgr2=0x7f7e601a6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 0x7f7e601a6c30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009980 tx=0x7f7e48032df0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 shutdown_connections 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7e4c0a2d00 0x7f7e4c0a51c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e6010ffc0 0x7f7e601adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e601090a0 0x7f7e601a6c30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.851+0000 7f7e64ff4640 1 --2- 172.21.15.16:0/640823269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e60108740 0x7f7e601a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.852+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 >> 172.21.15.16:0/640823269 conn(0x7f7e6006e7b0 msgr2=0x7f7e600703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.852+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 shutdown_connections 2024-04-25T01:35:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:07.852+0000 7f7e64ff4640 1 -- 172.21.15.16:0/640823269 wait complete. 2024-04-25T01:35:09.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:09 smithi016 ceph-mon[29153]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:09.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:09 smithi038 ceph-mon[32841]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:09.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:09 smithi156 ceph-mon[32631]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:11.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:11 smithi016 ceph-mon[29153]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 66 MiB/s wr, 1.10k op/s 2024-04-25T01:35:11.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:11 smithi038 ceph-mon[32841]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 66 MiB/s wr, 1.10k op/s 2024-04-25T01:35:11.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:11 smithi156 ceph-mon[32631]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 66 MiB/s wr, 1.10k op/s 2024-04-25T01:35:13.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:13 smithi016 ceph-mon[29153]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:13.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:13 smithi038 ceph-mon[32841]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:13.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:13 smithi156 ceph-mon[32631]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:15.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:15 smithi016 ceph-mon[29153]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 68 MiB/s wr, 1.14k op/s 2024-04-25T01:35:15.463 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:15 smithi038 ceph-mon[32841]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 68 MiB/s wr, 1.14k op/s 2024-04-25T01:35:15.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:15 smithi156 ceph-mon[32631]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 68 MiB/s wr, 1.14k op/s 2024-04-25T01:35:17.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:17 smithi016 ceph-mon[29153]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 56 MiB/s wr, 943 op/s 2024-04-25T01:35:17.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:17 smithi038 ceph-mon[32841]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 56 MiB/s wr, 943 op/s 2024-04-25T01:35:17.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:17 smithi156 ceph-mon[32631]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 56 MiB/s wr, 943 op/s 2024-04-25T01:35:19.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:19 smithi016 ceph-mon[29153]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 65 MiB/s wr, 1.09k op/s 2024-04-25T01:35:19.462 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:19 smithi038 ceph-mon[32841]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 65 MiB/s wr, 1.09k op/s 2024-04-25T01:35:19.497 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:19 smithi156 ceph-mon[32631]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 65 MiB/s wr, 1.09k op/s 2024-04-25T01:35:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 -- 172.21.15.16:0/2021337698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081053b0 msgr2=0x7fea0810bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 --2- 172.21.15.16:0/2021337698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081053b0 0x7fea0810bc40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009be0 tx=0x7fe9f802a6c0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 -- 172.21.15.16:0/2021337698 shutdown_connections 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 --2- 172.21.15.16:0/2021337698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081053b0 0x7fea0810bc40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 --2- 172.21.15.16:0/2021337698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea081049f0 0x7fea08104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 --2- 172.21.15.16:0/2021337698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea08103800 0x7fea08103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.010+0000 7fea0759e640 1 -- 172.21.15.16:0/2021337698 >> 172.21.15.16:0/2021337698 conn(0x7fea080fedb0 msgr2=0x7fea081011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.011+0000 7fea0759e640 1 -- 172.21.15.16:0/2021337698 shutdown_connections 2024-04-25T01:35:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.011+0000 7fea0759e640 1 -- 172.21.15.16:0/2021337698 wait complete. 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.011+0000 7fea0759e640 1 Processor -- start 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 -- start start 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea08103800 0x7fea0819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 0x7fea0819e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea081053b0 0x7fea081a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea0810f310 con 0x7fea081049f0 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea0810f190 con 0x7fea08103800 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0759e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea0810f490 con 0x7fea081053b0 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea0659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea08103800 0x7fea0819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea05d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 0x7fea0819e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea05d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 0x7fea0819e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59866/0 (socket says 172.21.15.16:59866) 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea05d9b640 1 -- 172.21.15.16:0/2637489893 learned_addr learned my addr 172.21.15.16:0/2637489893 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.012+0000 7fea06d9d640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea081053b0 0x7fea081a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea05d9b640 1 -- 172.21.15.16:0/2637489893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea081053b0 msgr2=0x7fea081a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea05d9b640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea081053b0 0x7fea081a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea05d9b640 1 -- 172.21.15.16:0/2637489893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea08103800 msgr2=0x7fea0819de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea05d9b640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea08103800 0x7fea0819de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea05d9b640 1 -- 172.21.15.16:0/2637489893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea081a5a20 con 0x7fea081049f0 2024-04-25T01:35:20.990 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea0659c640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea08103800 0x7fea0819de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.013+0000 7fea05d9b640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 0x7fea0819e340 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000e9c0 tx=0x7fe9f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.014+0000 7fe9ff7fe640 1 -- 172.21.15.16:0/2637489893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f000ccc0 con 0x7fea081049f0 2024-04-25T01:35:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.014+0000 7fe9ff7fe640 1 -- 172.21.15.16:0/2637489893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9f00054d0 con 0x7fea081049f0 2024-04-25T01:35:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.014+0000 7fe9ff7fe640 1 -- 172.21.15.16:0/2637489893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0010430 con 0x7fea081049f0 2024-04-25T01:35:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.014+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea081a5d10 con 0x7fea081049f0 2024-04-25T01:35:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.014+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea081a61b0 con 0x7fea081049f0 2024-04-25T01:35:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.017+0000 7fe9ff7fe640 1 -- 172.21.15.16:0/2637489893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe9f0004f00 con 0x7fea081049f0 2024-04-25T01:35:20.997 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.021+0000 7fea0fa53740 1 Processor -- start 2024-04-25T01:35:20.997 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.021+0000 7fe9ff7fe640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe9f40a2cc0 0x7fe9f40a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.997 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.021+0000 7fe9ff7fe640 1 -- 172.21.15.16:0/2637489893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe9f00cf4e0 con 0x7fea081049f0 2024-04-25T01:35:20.998 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 -- start start 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065dab030 0x55f065dab450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065db0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065db3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f065d0ecb0 con 0x55f065db0b90 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f065cb75b0 con 0x55f065daba20 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.022+0000 7fea0fa53740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f065d0ac70 con 0x55f065dab030 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065db3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065db3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59876/0 (socket says 172.21.15.16:59876) 2024-04-25T01:35:20.999 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea06d9d640 1 -- 172.21.15.16:0/3405438077 learned_addr learned my addr 172.21.15.16:0/3405438077 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:21.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea05d9b640 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065db0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea05d9b640 1 -- 172.21.15.16:0/3405438077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065dab030 msgr2=0x55f065dab450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:21.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea05d9b640 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065dab030 0x55f065dab450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea05d9b640 1 -- 172.21.15.16:0/3405438077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 msgr2=0x55f065db3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:21.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea05d9b640 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065db3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.000 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.023+0000 7fea05d9b640 1 -- 172.21.15.16:0/3405438077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f065a23210 con 0x55f065daba20 2024-04-25T01:35:21.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.024+0000 7fea05d9b640 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065db0520 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000c6a0 tx=0x7fe9f000efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b74d2bad79ab31dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:21.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.024+0000 7fea0659c640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe9f40a2cc0 0x7fe9f40a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.025+0000 7fe9fcff9640 1 -- 172.21.15.16:0/3405438077 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f009bcc0 con 0x55f065daba20 2024-04-25T01:35:21.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.025+0000 7fe9fcff9640 1 -- 172.21.15.16:0/3405438077 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9f00a0070 con 0x55f065daba20 2024-04-25T01:35:21.001 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.025+0000 7fe9fcff9640 1 -- 172.21.15.16:0/3405438077 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0005310 con 0x55f065daba20 2024-04-25T01:35:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 -- 172.21.15.16:0/3405438077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 msgr2=0x55f065db0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0659c640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe9f40a2cc0 0x7fe9f40a5180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55f065dac280 tx=0x7fea00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065db0520 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000c6a0 tx=0x7fe9f000efb0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 -- 172.21.15.16:0/3405438077 shutdown_connections 2024-04-25T01:35:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065db3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065db0520 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 --2- 172.21.15.16:0/3405438077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065dab030 0x55f065dab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 -- 172.21.15.16:0/3405438077 >> 172.21.15.16:0/3405438077 conn(0x55f065da66a0 msgr2=0x55f065da6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 -- 172.21.15.16:0/3405438077 shutdown_connections 2024-04-25T01:35:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 -- 172.21.15.16:0/3405438077 wait complete. 2024-04-25T01:35:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.026+0000 7fea0fa53740 1 -- start start 2024-04-25T01:35:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 Processor -- start 2024-04-25T01:35:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065db54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065db5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f065db3ae0 con 0x55f065db0b90 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f065ccab50 con 0x55f065dab030 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f065db3c60 con 0x55f065daba20 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0fa53740 10 client.0 _ll_register_callbacks cb 0x55f065d05ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea05d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065db5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea0659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065db54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea05d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065db5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45212/0 (socket says 172.21.15.16:45212) 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59888/0 (socket says 172.21.15.16:59888) 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.027+0000 7fea05d9b640 1 -- 172.21.15.16:0/428126876 learned_addr learned my addr 172.21.15.16:0/428126876 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea05d9b640 1 -- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 msgr2=0x55f065db54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea05d9b640 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065db54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea05d9b640 1 -- 172.21.15.16:0/428126876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 msgr2=0x55f065dbca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea05d9b640 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea05d9b640 1 -- 172.21.15.16:0/428126876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f065aef620 con 0x55f065daba20 2024-04-25T01:35:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea0659c640 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065db54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fea05d9b640 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065db5a20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0098690 tx=0x7fe9f0098010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83cafc2277b4dc34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fe9edffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:35:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.028+0000 7fe9edffb640 1 -- 172.21.15.16:0/428126876 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00b8030 con 0x55f065daba20 2024-04-25T01:35:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.029+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f00a0070 con 0x55f065daba20 2024-04-25T01:35:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.029+0000 7fe9edffb640 1 -- 172.21.15.16:0/428126876 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe9f0099fc0 con 0x55f065daba20 2024-04-25T01:35:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.029+0000 7fe9edffb640 1 -- 172.21.15.16:0/428126876 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe9f00052f0 con 0x55f065daba20 2024-04-25T01:35:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.029+0000 7fea0fa53740 10 client.25507 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.029+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f065db3ba0 con 0x55f065daba20 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.029+0000 7fe9edffb640 1 -- 172.21.15.16:0/428126876 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9f00a0070 con 0x55f065daba20 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fe9edffb640 1 -- 172.21.15.16:0/428126876 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe9f00a4b80 con 0x55f065daba20 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fea0fa53740 10 client.25507 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fea0fa53740 10 client.25507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fea0fa53740 20 client.25507 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fea0fa53740 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065dd20a0 0x55f065dd4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fea0fa53740 4 client.25507 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.030+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f065d1c1f0 con 0x55f065dd20a0 2024-04-25T01:35:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.031+0000 7fea0659c640 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065dd20a0 0x55f065dd4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.033+0000 7fe9edffb640 10 client.25507 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.034+0000 7fea0659c640 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065dd20a0 0x55f065dd4490 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25d154a70d0af4e server_cookie=28851710d146c921 in_seq=0 out_seq=0 2024-04-25T01:35:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.035+0000 7fe9edffb640 1 -- 172.21.15.16:0/428126876 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f065d1c1f0 con 0x55f065dd20a0 2024-04-25T01:35:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.035+0000 7fe9edffb640 10 client.25507 handle_command_reply: tid=0 2024-04-25T01:35:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.035+0000 7fea0fa53740 1 client.25507 shutdown 2024-04-25T01:35:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.035+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 msgr2=0x55f065db5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.035+0000 7fea0fa53740 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065db5a20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0098690 tx=0x7fe9f0098010 comp rx=0 tx=0).stop 2024-04-25T01:35:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 shutdown_connections 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065db5a20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065dd20a0 0x55f065dd4490 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 --2- 172.21.15.16:0/428126876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065db54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 >> 172.21.15.16:0/428126876 conn(0x55f065da66a0 msgr2=0x55f065da58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fe9edffb640 0 client.25507 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 shutdown_connections 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 1 -- 172.21.15.16:0/428126876 wait complete. 2024-04-25T01:35:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.036+0000 7fea0fa53740 20 client.25507 trim_cache size 0 max 16384 2024-04-25T01:35:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.066+0000 7fea0fa53740 1 Processor -- start 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.066+0000 7fea0fa53740 1 -- start start 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.066+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065dab030 0x55f065da1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065dbca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 0x55f065da5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea0fa53740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea081a92d0 con 0x55f065dab030 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea0fa53740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea081a6ba0 con 0x55f065daba20 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea0fa53740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea08071f90 con 0x55f065db0b90 2024-04-25T01:35:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 0x55f065da5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 0x55f065da5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45222/0 (socket says 172.21.15.16:45222) 2024-04-25T01:35:21.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.067+0000 7fea06d9d640 1 -- 172.21.15.16:0/1125882333 learned_addr learned my addr 172.21.15.16:0/1125882333 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:21.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.069+0000 7fea0659c640 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065dab030 0x55f065da1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.069+0000 7fea06d9d640 1 -- 172.21.15.16:0/1125882333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 msgr2=0x55f065dbca20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:21.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.069+0000 7fea06d9d640 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065dbca20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.069+0000 7fea06d9d640 1 -- 172.21.15.16:0/1125882333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065dab030 msgr2=0x55f065da1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:21.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.069+0000 7fea06d9d640 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065dab030 0x55f065da1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.069+0000 7fea06d9d640 1 -- 172.21.15.16:0/1125882333 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f065dabe00 con 0x55f065db0b90 2024-04-25T01:35:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.070+0000 7fea06d9d640 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 0x55f065da5010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009be0 tx=0x7fe9f8030670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58e8c28b4ed7009f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.070+0000 7fe9effff640 1 -- 172.21.15.16:0/1125882333 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8036040 con 0x55f065db0b90 2024-04-25T01:35:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.071+0000 7fe9effff640 1 -- 172.21.15.16:0/1125882333 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9f8037040 con 0x55f065db0b90 2024-04-25T01:35:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.071+0000 7fe9effff640 1 -- 172.21.15.16:0/1125882333 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8004540 con 0x55f065db0b90 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 -- 172.21.15.16:0/1125882333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 msgr2=0x55f065da5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 0x55f065da5010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009be0 tx=0x7fe9f8030670 comp rx=0 tx=0).stop 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 -- 172.21.15.16:0/1125882333 shutdown_connections 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065db0b90 0x55f065da5010 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065daba20 0x55f065dbca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 --2- 172.21.15.16:0/1125882333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065dab030 0x55f065da1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.049 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 -- 172.21.15.16:0/1125882333 >> 172.21.15.16:0/1125882333 conn(0x55f065da66a0 msgr2=0x55f065dacee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 -- 172.21.15.16:0/1125882333 shutdown_connections 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 -- 172.21.15.16:0/1125882333 wait complete. 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 -- start start 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.072+0000 7fea0fa53740 1 Processor -- start 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065dba0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065dba620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea08072600 con 0x55f065db0b90 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea081a64d0 con 0x55f065dab030 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea08072cd0 con 0x55f065daba20 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0fa53740 10 client.0 _ll_register_callbacks cb 0x7fe9f0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea05d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065dba620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.050 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59904/0 (socket says 172.21.15.16:59904) 2024-04-25T01:35:21.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 -- 172.21.15.16:0/2629712635 learned_addr learned my addr 172.21.15.16:0/2629712635 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:21.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea0659c640 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065dba0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 -- 172.21.15.16:0/2629712635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 msgr2=0x55f065dba620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:21.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065dba620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 -- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 msgr2=0x55f065dba0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065dba0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.073+0000 7fea06d9d640 1 -- 172.21.15.16:0/2629712635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f065dbbbb0 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.074+0000 7fea06d9d640 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbb5c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8030850 tx=0x7fe9f8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b6fbf815d57608 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.074+0000 7fe9edffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.074+0000 7fe9edffb640 1 -- 172.21.15.16:0/2629712635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8055060 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.075+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f8037040 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.075+0000 7fe9edffb640 1 -- 172.21.15.16:0/2629712635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe9f8004280 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.075+0000 7fe9edffb640 1 -- 172.21.15.16:0/2629712635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe9f8036040 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.075+0000 7fea0fa53740 10 client.15873 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.075+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f065e385f0 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fe9edffb640 1 -- 172.21.15.16:0/2629712635 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9f8037040 con 0x55f065db0b90 2024-04-25T01:35:21.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fe9edffb640 1 -- 172.21.15.16:0/2629712635 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fe9f8046070 con 0x55f065db0b90 2024-04-25T01:35:21.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fea0fa53740 10 client.15873 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:21.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fea0fa53740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:21.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fea0fa53740 20 client.15873 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:21.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fea0fa53740 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065e38ff0 0x55f065e3b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:21.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fea0fa53740 4 client.15873 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:35:21.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.076+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f065d16540 con 0x55f065e38ff0 2024-04-25T01:35:21.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.077+0000 7fea0659c640 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065e38ff0 0x55f065e3b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:21.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.077+0000 7fea0659c640 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065e38ff0 0x55f065e3b410 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d20765a0ff282b77 server_cookie=f070c61a35fc5a48 in_seq=0 out_seq=0 2024-04-25T01:35:21.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.077+0000 7fe9edffb640 10 client.15873 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fe9edffb640 1 -- 172.21.15.16:0/2629712635 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f065d16540 con 0x55f065e38ff0 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fe9edffb640 10 client.15873 handle_command_reply: tid=0 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 client.15873 shutdown 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 msgr2=0x55f065dbb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbb5c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8030850 tx=0x7fe9f8003ba0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 shutdown_connections 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f065db0b90 0x55f065dbb5c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f065daba20 0x55f065dba620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f065e38ff0 0x55f065e3b410 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 --2- 172.21.15.16:0/2629712635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f065dab030 0x55f065dba0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 >> 172.21.15.16:0/2629712635 conn(0x55f065da66a0 msgr2=0x55f065da9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.084+0000 7fe9edffb640 0 client.15873 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:21.061 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.085+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 shutdown_connections 2024-04-25T01:35:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.085+0000 7fea0fa53740 1 -- 172.21.15.16:0/2629712635 wait complete. 2024-04-25T01:35:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.085+0000 7fea0fa53740 20 client.15873 trim_cache size 0 max 16384 2024-04-25T01:35:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:35:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:35:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:35:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:35:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe9f40a2cc0 msgr2=0x7fe9f40a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe9f40a2cc0 0x7fe9f40a5180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55f065dac280 tx=0x7fea00005f90 comp rx=0 tx=0).stop 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 msgr2=0x7fea0819e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 0x7fea0819e340 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000e9c0 tx=0x7fe9f000ee80 comp rx=0 tx=0).stop 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 shutdown_connections 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fe9f40a2cc0 0x7fe9f40a5180 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55f065dac280 tx=0x7fea00005f90 comp rx=0 tx=0).stop 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea081053b0 0x7fea081a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea081049f0 0x7fea0819e340 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 --2- 172.21.15.16:0/2637489893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea08103800 0x7fea0819de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.087+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 >> 172.21.15.16:0/2637489893 conn(0x7fea080fedb0 msgr2=0x7fea08077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.089+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 shutdown_connections 2024-04-25T01:35:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:21.089+0000 7fea0759e640 1 -- 172.21.15.16:0/2637489893 wait complete. 2024-04-25T01:35:21.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:21 smithi016 ceph-mon[29153]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 65 MiB/s wr, 1.09k op/s 2024-04-25T01:35:21.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:21 smithi038 ceph-mon[32841]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 65 MiB/s wr, 1.09k op/s 2024-04-25T01:35:21.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:21 smithi156 ceph-mon[32631]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 65 MiB/s wr, 1.09k op/s 2024-04-25T01:35:23.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:23 smithi016 ceph-mon[29153]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:23.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:23 smithi038 ceph-mon[32841]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:23.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:23 smithi156 ceph-mon[32631]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:25.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:25 smithi016 ceph-mon[29153]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:25.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:25 smithi038 ceph-mon[32841]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:25.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:25 smithi156 ceph-mon[32631]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:27.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:27 smithi016 ceph-mon[29153]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 56 MiB/s wr, 952 op/s 2024-04-25T01:35:27.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:27 smithi038 ceph-mon[32841]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 56 MiB/s wr, 952 op/s 2024-04-25T01:35:27.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:27 smithi156 ceph-mon[32631]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 56 MiB/s wr, 952 op/s 2024-04-25T01:35:29.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:29 smithi016 ceph-mon[29153]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:29.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:29 smithi038 ceph-mon[32841]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:29.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:29 smithi156 ceph-mon[32631]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 66 MiB/s wr, 1.11k op/s 2024-04-25T01:35:31.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:31 smithi016 ceph-mon[29153]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:31.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:31 smithi038 ceph-mon[32841]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:31.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:31 smithi156 ceph-mon[32631]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:33.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:33 smithi016 ceph-mon[29153]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:33.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:33 smithi038 ceph-mon[32841]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:33.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:33 smithi156 ceph-mon[32631]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:34.224 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.248+0000 7f4f442e7640 1 -- 172.21.15.16:0/390191171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c10ffc0 msgr2=0x7f4f3c1123b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.248+0000 7f4f442e7640 1 --2- 172.21.15.16:0/390191171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c10ffc0 0x7f4f3c1123b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009ca0 tx=0x7f4f3402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.249+0000 7f4f442e7640 1 -- 172.21.15.16:0/390191171 shutdown_connections 2024-04-25T01:35:34.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.249+0000 7f4f442e7640 1 --2- 172.21.15.16:0/390191171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c10ffc0 0x7f4f3c1123b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.249+0000 7f4f442e7640 1 --2- 172.21.15.16:0/390191171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c10fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.249+0000 7f4f442e7640 1 --2- 172.21.15.16:0/390191171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f3c108740 0x7f4f3c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.225 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.249+0000 7f4f442e7640 1 -- 172.21.15.16:0/390191171 >> 172.21.15.16:0/390191171 conn(0x7f4f3c06e7b0 msgr2=0x7f4f3c070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.250+0000 7f4f442e7640 1 -- 172.21.15.16:0/390191171 shutdown_connections 2024-04-25T01:35:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.250+0000 7f4f442e7640 1 -- 172.21.15.16:0/390191171 wait complete. 2024-04-25T01:35:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 Processor -- start 2024-04-25T01:35:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 -- start start 2024-04-25T01:35:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c108740 0x7f4f3c1a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f3c10ffc0 0x7f4f3c1a7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c115ae0 con 0x7f4f3c108740 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c115960 con 0x7f4f3c1090a0 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f442e7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c115c60 con 0x7f4f3c10ffc0 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f41882640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f42083640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c108740 0x7f4f3c1a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f41882640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c1a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:37896/0 (socket says 172.21.15.16:37896) 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f41882640 1 -- 172.21.15.16:0/1558014389 learned_addr learned my addr 172.21.15.16:0/1558014389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.251+0000 7f4f42884640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f3c10ffc0 0x7f4f3c1a7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f41882640 1 -- 172.21.15.16:0/1558014389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f3c10ffc0 msgr2=0x7f4f3c1a7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f41882640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f3c10ffc0 0x7f4f3c1a7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f41882640 1 -- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c108740 msgr2=0x7f4f3c1a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f41882640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c108740 0x7f4f3c1a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f41882640 1 -- 172.21.15.16:0/1558014389 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c109980 con 0x7f4f3c1090a0 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f42083640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c108740 0x7f4f3c1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f41882640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c1a05a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00d950 tx=0x7f4f2c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f337fe640 1 -- 172.21.15.16:0/1558014389 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c017070 con 0x7f4f3c1090a0 2024-04-25T01:35:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.252+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c109c70 con 0x7f4f3c1090a0 2024-04-25T01:35:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.253+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c10a210 con 0x7f4f3c1090a0 2024-04-25T01:35:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.253+0000 7f4f337fe640 1 -- 172.21.15.16:0/1558014389 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f2c009c30 con 0x7f4f3c1090a0 2024-04-25T01:35:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.253+0000 7f4f337fe640 1 -- 172.21.15.16:0/1558014389 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00f620 con 0x7f4f3c1090a0 2024-04-25T01:35:34.232 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.256+0000 7f4f46ff1740 1 Processor -- start 2024-04-25T01:35:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 -- start start 2024-04-25T01:35:34.233 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e39a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e3eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e41650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624a3d94cb0 con 0x5624a3e39660 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5624a3d3d5b0 con 0x5624a3e3a050 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.257+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5624a3d90c70 con 0x5624a3e3f1c0 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e39a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.234 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e39a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49888/0 (socket says 172.21.15.16:49888) 2024-04-25T01:35:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 -- 172.21.15.16:0/3429749059 learned_addr learned my addr 172.21.15.16:0/3429749059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 -- 172.21.15.16:0/3429749059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 msgr2=0x5624a3e41650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e41650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 -- 172.21.15.16:0/3429749059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 msgr2=0x5624a3e3eb50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e3eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.258+0000 7f4f42884640 1 -- 172.21.15.16:0/3429749059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a3aa9210 con 0x5624a3e39660 2024-04-25T01:35:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.259+0000 7f4f42884640 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e39a80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4f34031a50 tx=0x7f4f34005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb468d5a3cbdff8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.259+0000 7f4f30bfe640 1 -- 172.21.15.16:0/3429749059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3402d070 con 0x5624a3e39660 2024-04-25T01:35:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.259+0000 7f4f30bfe640 1 -- 172.21.15.16:0/3429749059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f34036070 con 0x5624a3e39660 2024-04-25T01:35:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.260+0000 7f4f30bfe640 1 -- 172.21.15.16:0/3429749059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34004040 con 0x5624a3e39660 2024-04-25T01:35:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.260+0000 7f4f337fe640 1 -- 172.21.15.16:0/1558014389 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4f2c00f7c0 con 0x7f4f3c1090a0 2024-04-25T01:35:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.260+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3429749059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 msgr2=0x5624a3e39a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.260+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e39a80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4f34031a50 tx=0x7f4f34005bd0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.261+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3429749059 shutdown_connections 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.261+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e41650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.261+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e3eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.261+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3429749059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e39a80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.261+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3429749059 >> 172.21.15.16:0/3429749059 conn(0x5624a3e2c4c0 msgr2=0x5624a3e2c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.261+0000 7f4f337fe640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4f280a2d30 0x7f4f280a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.262+0000 7f4f337fe640 1 -- 172.21.15.16:0/1558014389 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4f2c0ceb00 con 0x7f4f3c1090a0 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.262+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3429749059 shutdown_connections 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.262+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3429749059 wait complete. 2024-04-25T01:35:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.262+0000 7f4f42083640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4f280a2d30 0x7f4f280a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 -- start start 2024-04-25T01:35:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f42083640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4f280a2d30 0x7f4f280a51f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x5624a3e2d430 tx=0x7f4f3800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 Processor -- start 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 0x5624a3e43a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e3a050 0x5624a3e43fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e4aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624a3e421b0 con 0x5624a3e3a050 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5624a3d50b50 con 0x5624a3e39660 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5624a3e42330 con 0x5624a3e3f1c0 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f42083640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 0x5624a3e43a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f42083640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 0x5624a3e43a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:37920/0 (socket says 172.21.15.16:37920) 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f42083640 1 -- 172.21.15.16:0/1088755009 learned_addr learned my addr 172.21.15.16:0/1088755009 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42884640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e4aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f41882640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e3a050 0x5624a3e43fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42083640 1 -- 172.21.15.16:0/1088755009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 msgr2=0x5624a3e4aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42083640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e4aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42083640 1 -- 172.21.15.16:0/1088755009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e3a050 msgr2=0x5624a3e43fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42083640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e3a050 0x5624a3e43fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42083640 1 -- 172.21.15.16:0/1088755009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a3b75620 con 0x5624a3e39660 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.263+0000 7f4f46ff1740 10 client.0 _ll_register_callbacks cb 0x5624a3d8bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f42083640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 0x5624a3e43a70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005940 tx=0x7f4f38004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91c5e2c4e640bd04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f34036070 con 0x5624a3e39660 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f25ffb640 1 -- 172.21.15.16:0/1088755009 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38016070 con 0x5624a3e39660 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f25ffb640 1 -- 172.21.15.16:0/1088755009 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4f38027480 con 0x5624a3e39660 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.264+0000 7f4f25ffb640 1 -- 172.21.15.16:0/1088755009 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4f38027680 con 0x5624a3e39660 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 10 client.25562 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=22}) -- 0x5624a3e4e350 con 0x5624a3e39660 2024-04-25T01:35:34.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f25ffb640 1 -- 172.21.15.16:0/1088755009 <== mon.2 v2:172.21.15.38:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4f38027820 con 0x5624a3e39660 2024-04-25T01:35:34.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 10 client.25562 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:34.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 10 client.25562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 20 client.25562 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3e4fe50 0x5624a3e522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 4 client.25562 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4f34036070 con 0x5624a3e4fe50 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.265+0000 7f4f41882640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3e4fe50 0x5624a3e522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.266+0000 7f4f41882640 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3e4fe50 0x5624a3e522f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b9b2f0fe3f12898 server_cookie=bae73531571081b7 in_seq=0 out_seq=0 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.266+0000 7f4f25ffb640 10 client.25562 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.266+0000 7f4f25ffb640 1 -- 172.21.15.16:0/1088755009 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f4f34036070 con 0x5624a3e4fe50 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.266+0000 7f4f25ffb640 10 client.25562 handle_command_reply: tid=0 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.266+0000 7f4f46ff1740 1 client.25562 shutdown 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 msgr2=0x5624a3e43a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 0x5624a3e43a70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005940 tx=0x7f4f38004730 comp rx=0 tx=0).stop 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 shutdown_connections 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3e4aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e3a050 0x5624a3e43fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3e4fe50 0x5624a3e522f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1088755009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e39660 0x5624a3e43a70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 >> 172.21.15.16:0/1088755009 conn(0x5624a3e2c4c0 msgr2=0x5624a3e2b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f25ffb640 0 client.25562 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 shutdown_connections 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.267+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1088755009 wait complete. 2024-04-25T01:35:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.268+0000 7f4f46ff1740 20 client.25562 trim_cache size 0 max 16384 2024-04-25T01:35:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 Processor -- start 2024-04-25T01:35:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 -- start start 2024-04-25T01:35:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e2c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3a050 0x5624a3e4afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3f1c0 0x5624a3e2f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f20010ea0 con 0x5624a3e39660 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2000e770 con 0x5624a3e3f1c0 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2000c5c0 con 0x5624a3e3a050 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f42083640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e2c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f42083640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e2c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49908/0 (socket says 172.21.15.16:49908) 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f42083640 1 -- 172.21.15.16:0/1312748121 learned_addr learned my addr 172.21.15.16:0/1312748121 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.299+0000 7f4f42884640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3f1c0 0x5624a3e2f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f42884640 1 -- 172.21.15.16:0/1312748121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3a050 msgr2=0x5624a3e4afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f41882640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3a050 0x5624a3e4afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f42884640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3a050 0x5624a3e4afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f42884640 1 -- 172.21.15.16:0/1312748121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 msgr2=0x5624a3e2c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f42884640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e2c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f42884640 1 -- 172.21.15.16:0/1312748121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a3e3e620 con 0x5624a3e3f1c0 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f42083640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e2c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.300+0000 7f4f41882640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3a050 0x5624a3e4afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.301+0000 7f4f42884640 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3f1c0 0x5624a3e2f550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009ca0 tx=0x7f4f34000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d43c6bcec4a58107 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.301+0000 7f4f27fff640 1 -- 172.21.15.16:0/1312748121 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34003b70 con 0x5624a3e3f1c0 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.301+0000 7f4f27fff640 1 -- 172.21.15.16:0/1312748121 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f34036070 con 0x5624a3e3f1c0 2024-04-25T01:35:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.301+0000 7f4f27fff640 1 -- 172.21.15.16:0/1312748121 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34030840 con 0x5624a3e3f1c0 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1312748121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3f1c0 msgr2=0x5624a3e2f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3f1c0 0x5624a3e2f550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009ca0 tx=0x7f4f34000f40 comp rx=0 tx=0).stop 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1312748121 shutdown_connections 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3f1c0 0x5624a3e2f550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3a050 0x5624a3e4afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/1312748121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e2c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.302+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1312748121 >> 172.21.15.16:0/1312748121 conn(0x5624a3e2c4c0 msgr2=0x5624a3e3a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.303+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1312748121 shutdown_connections 2024-04-25T01:35:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.303+0000 7f4f46ff1740 1 -- 172.21.15.16:0/1312748121 wait complete. 2024-04-25T01:35:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 -- start start 2024-04-25T01:35:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 Processor -- start 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e48030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e48570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3eb4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2000cc30 con 0x5624a3e39660 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2000e0a0 con 0x5624a3e3a050 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2000d300 con 0x5624a3e3f1c0 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f46ff1740 10 client.0 _ll_register_callbacks cb 0x7f4f3800aae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f41882640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e48570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.304+0000 7f4f42083640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e48030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e48030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49910/0 (socket says 172.21.15.16:49910) 2024-04-25T01:35:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 -- 172.21.15.16:0/3569791055 learned_addr learned my addr 172.21.15.16:0/3569791055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 -- 172.21.15.16:0/3569791055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 msgr2=0x5624a3eb4460 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3eb4460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 -- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 msgr2=0x5624a3e48570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e48570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 -- 172.21.15.16:0/3569791055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a3eb4a20 con 0x5624a3e39660 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.305+0000 7f4f42083640 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e48030 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005940 tx=0x7f4f38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5a2f6dac58c9a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.306+0000 7f4f25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.306+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3569791055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38042060 con 0x5624a3e39660 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.306+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f34036070 con 0x5624a3e39660 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.306+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3569791055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4f3801c630 con 0x5624a3e39660 2024-04-25T01:35:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.306+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3569791055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4f38016040 con 0x5624a3e39660 2024-04-25T01:35:34.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.310+0000 7f4f46ff1740 10 client.15891 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:34.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.310+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5624a3eb5d10 con 0x5624a3e39660 2024-04-25T01:35:34.290 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3569791055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f34036070 con 0x5624a3e39660 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3569791055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4f3803d8d0 con 0x5624a3e39660 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f46ff1740 10 client.15891 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f46ff1740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f46ff1740 20 client.15891 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3eb66f0 0x5624a3eb8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f46ff1740 4 client.15891 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624a3da21f0 con 0x5624a3eb66f0 2024-04-25T01:35:34.291 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.314+0000 7f4f41882640 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3eb66f0 0x5624a3eb8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.315+0000 7f4f41882640 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3eb66f0 0x5624a3eb8b10 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5bf3ea04656b174 server_cookie=2c342732f7231e62 in_seq=0 out_seq=0 2024-04-25T01:35:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.315+0000 7f4f25ffb640 10 client.15891 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.318+0000 7f4f25ffb640 1 -- 172.21.15.16:0/3569791055 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5624a3da21f0 con 0x5624a3eb66f0 2024-04-25T01:35:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.318+0000 7f4f25ffb640 10 client.15891 handle_command_reply: tid=0 2024-04-25T01:35:34.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.318+0000 7f4f46ff1740 1 client.15891 shutdown 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 msgr2=0x5624a3e48030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e48030 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005940 tx=0x7f4f38017000 comp rx=0 tx=0).stop 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 shutdown_connections 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624a3e3f1c0 0x5624a3eb4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5624a3eb66f0 0x5624a3eb8b10 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5624a3e3a050 0x5624a3e48570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 --2- 172.21.15.16:0/3569791055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624a3e39660 0x5624a3e48030 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 >> 172.21.15.16:0/3569791055 conn(0x5624a3e2c4c0 msgr2=0x5624a3e2c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f25ffb640 0 client.15891 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 shutdown_connections 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.319+0000 7f4f46ff1740 1 -- 172.21.15.16:0/3569791055 wait complete. 2024-04-25T01:35:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.320+0000 7f4f46ff1740 20 client.15891 trim_cache size 0 max 16384 2024-04-25T01:35:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:35:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:35:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:35:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:35:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4f280a2d30 msgr2=0x7f4f280a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4f280a2d30 0x7f4f280a51f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x5624a3e2d430 tx=0x7f4f3800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.298 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 msgr2=0x7f4f3c1a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c1a05a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00d950 tx=0x7f4f2c00de10 comp rx=0 tx=0).stop 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 shutdown_connections 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f4f280a2d30 0x7f4f280a51f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f3c10ffc0 0x7f4f3c1a7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f3c1090a0 0x7f4f3c1a05a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 --2- 172.21.15.16:0/1558014389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f3c108740 0x7f4f3c1a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 >> 172.21.15.16:0/1558014389 conn(0x7f4f3c06e7b0 msgr2=0x7f4f3c070310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 shutdown_connections 2024-04-25T01:35:34.299 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:34.321+0000 7f4f442e7640 1 -- 172.21.15.16:0/1558014389 wait complete. 2024-04-25T01:35:35.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:35 smithi016 ceph-mon[29153]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:35.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:35 smithi038 ceph-mon[32841]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:35.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:35 smithi156 ceph-mon[32631]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 68 MiB/s wr, 1.15k op/s 2024-04-25T01:35:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:37 smithi016 ceph-mon[29153]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 56 MiB/s wr, 947 op/s 2024-04-25T01:35:37.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:37 smithi038 ceph-mon[32841]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 56 MiB/s wr, 947 op/s 2024-04-25T01:35:37.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:37 smithi156 ceph-mon[32631]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 56 MiB/s wr, 947 op/s 2024-04-25T01:35:39.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:39 smithi016 ceph-mon[29153]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:39.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:39 smithi038 ceph-mon[32841]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:39.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:39 smithi156 ceph-mon[32631]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:41.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:41 smithi016 ceph-mon[29153]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:41.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:41 smithi038 ceph-mon[32841]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:41.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:41 smithi156 ceph-mon[32631]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 65 MiB/s wr, 1.10k op/s 2024-04-25T01:35:43.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:43 smithi016 ceph-mon[29153]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:43.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:43 smithi038 ceph-mon[32841]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:43.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:43 smithi156 ceph-mon[32631]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 63 MiB/s wr, 1.06k op/s 2024-04-25T01:35:45.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:45 smithi016 ceph-mon[29153]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 67 MiB/s wr, 1.14k op/s 2024-04-25T01:35:45.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:45 smithi038 ceph-mon[32841]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 67 MiB/s wr, 1.14k op/s 2024-04-25T01:35:45.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:45 smithi156 ceph-mon[32631]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 67 MiB/s wr, 1.14k op/s 2024-04-25T01:35:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.481+0000 7fc0637fe640 1 -- 172.21.15.16:0/3309248856 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c031cc0 con 0x7fc0641006a0 2024-04-25T01:35:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 -- 172.21.15.16:0/3309248856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0641006a0 msgr2=0x7fc064100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 --2- 172.21.15.16:0/3309248856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0641006a0 0x7fc064100ac0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009ca0 tx=0x7fc05c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 -- 172.21.15.16:0/3309248856 shutdown_connections 2024-04-25T01:35:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 --2- 172.21.15.16:0/3309248856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc064102250 0x7fc064108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 --2- 172.21.15.16:0/3309248856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064101890 0x7fc064101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 --2- 172.21.15.16:0/3309248856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0641006a0 0x7fc064100ac0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 -- 172.21.15.16:0/3309248856 >> 172.21.15.16:0/3309248856 conn(0x7fc0640fbc10 msgr2=0x7fc0640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.482+0000 7fc06af29640 1 -- 172.21.15.16:0/3309248856 shutdown_connections 2024-04-25T01:35:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.483+0000 7fc06af29640 1 -- 172.21.15.16:0/3309248856 wait complete. 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.483+0000 7fc06af29640 1 Processor -- start 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.483+0000 7fc06af29640 1 -- start start 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc06af29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0641006a0 0x7fc06419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc06af29640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc064101890 0x7fc06419e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc06af29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 0x7fc0641a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc06af29640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc06410a780 con 0x7fc064102250 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc06af29640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc06410a600 con 0x7fc0641006a0 2024-04-25T01:35:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc06af29640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc06410a900 con 0x7fc064101890 2024-04-25T01:35:47.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc0694c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 0x7fc0641a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc0694c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 0x7fc0641a5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42894/0 (socket says 172.21.15.16:42894) 2024-04-25T01:35:47.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc0694c6640 1 -- 172.21.15.16:0/3805744453 learned_addr learned my addr 172.21.15.16:0/3805744453 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:47.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc068cc5640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0641006a0 0x7fc06419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.461 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.484+0000 7fc063fff640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc064101890 0x7fc06419e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc0694c6640 1 -- 172.21.15.16:0/3805744453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc064101890 msgr2=0x7fc06419e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc0694c6640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc064101890 0x7fc06419e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc0694c6640 1 -- 172.21.15.16:0/3805744453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0641006a0 msgr2=0x7fc06419df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc0694c6640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0641006a0 0x7fc06419df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc0694c6640 1 -- 172.21.15.16:0/3805744453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0641a5af0 con 0x7fc064102250 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc0694c6640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 0x7fc0641a5480 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc05800d750 tx=0x7fc05800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc061ffb640 1 -- 172.21.15.16:0/3805744453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058017070 con 0x7fc064102250 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc061ffb640 1 -- 172.21.15.16:0/3805744453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0580097c0 con 0x7fc064102250 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc061ffb640 1 -- 172.21.15.16:0/3805744453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058004b80 con 0x7fc064102250 2024-04-25T01:35:47.462 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.485+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0641a5de0 con 0x7fc064102250 2024-04-25T01:35:47.463 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.486+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0641a6250 con 0x7fc064102250 2024-04-25T01:35:47.465 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc061ffb640 1 -- 172.21.15.16:0/3805744453 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc058004d20 con 0x7fc064102250 2024-04-25T01:35:47.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 Processor -- start 2024-04-25T01:35:47.466 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 -- start start 2024-04-25T01:35:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d8f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d8fe340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d900e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55592d85ccb0 con 0x55592d8f8e50 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55592d8055b0 con 0x55592d8fe9b0 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc06dc36740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55592d858c70 con 0x55592d8f9840 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc061ffb640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc04c0a2d00 0x7fc04c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc063fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d8f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc063fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d8f9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42904/0 (socket says 172.21.15.16:42904) 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.490+0000 7fc063fff640 1 -- 172.21.15.16:0/2044152185 learned_addr learned my addr 172.21.15.16:0/2044152185 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d8fe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc061ffb640 1 -- 172.21.15.16:0/3805744453 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc0580ce930 con 0x7fc064102250 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc063fff640 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d900e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 -- 172.21.15.16:0/2044152185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 msgr2=0x55592d900e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d900e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 -- 172.21.15.16:0/2044152185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 msgr2=0x55592d8f9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d8f9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 -- 172.21.15.16:0/2044152185 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592d571210 con 0x55592d8f9840 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d8fe340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc05c02aed0 tx=0x7fc05c009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56a5937446529ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.491+0000 7fc0694c6640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc04c0a2d00 0x7fc04c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.492+0000 7fc04affd640 1 -- 172.21.15.16:0/2044152185 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c031bc0 con 0x55592d8f9840 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.492+0000 7fc04affd640 1 -- 172.21.15.16:0/2044152185 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc05c036040 con 0x55592d8f9840 2024-04-25T01:35:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.492+0000 7fc04affd640 1 -- 172.21.15.16:0/2044152185 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0046c0 con 0x55592d8f9840 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc0694c6640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc04c0a2d00 0x7fc04c0a51c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc0580975d0 tx=0x7fc0580bb040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 -- 172.21.15.16:0/2044152185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 msgr2=0x55592d8fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d8fe340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc05c02aed0 tx=0x7fc05c009a20 comp rx=0 tx=0).stop 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 -- 172.21.15.16:0/2044152185 shutdown_connections 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d900e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d8fe340 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2044152185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d8f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 -- 172.21.15.16:0/2044152185 >> 172.21.15.16:0/2044152185 conn(0x55592d8f44c0 msgr2=0x55592d8f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.493+0000 7fc06dc36740 1 -- 172.21.15.16:0/2044152185 shutdown_connections 2024-04-25T01:35:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.497+0000 7fc06dc36740 1 -- 172.21.15.16:0/2044152185 wait complete. 2024-04-25T01:35:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.498+0000 7fc06dc36740 1 -- start start 2024-04-25T01:35:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.498+0000 7fc06dc36740 1 Processor -- start 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.499+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d903040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8f9840 0x55592d903580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8fe9b0 0x55592d90a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc06dc36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55592d901900 con 0x55592d8f8e50 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc06dc36740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55592d818b50 con 0x55592d8f9840 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc06dc36740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55592d901a80 con 0x55592d8fe9b0 2024-04-25T01:35:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc06dc36740 10 client.0 _ll_register_callbacks cb 0x55592d853ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d903040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d903040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42908/0 (socket says 172.21.15.16:42908) 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 -- 172.21.15.16:0/1780208619 learned_addr learned my addr 172.21.15.16:0/1780208619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8fe9b0 0x55592d90a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 -- 172.21.15.16:0/1780208619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8fe9b0 msgr2=0x55592d90a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8fe9b0 0x55592d90a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 -- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8f9840 msgr2=0x55592d903580 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8f9840 0x55592d903580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 -- 172.21.15.16:0/1780208619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592d63d620 con 0x55592d8f8e50 2024-04-25T01:35:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.500+0000 7fc063fff640 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d903040 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc05400af60 tx=0x7fc05400a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb74f63f89f4cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.501+0000 7fc043fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:35:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.501+0000 7fc043fff640 1 -- 172.21.15.16:0/1780208619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05401c040 con 0x55592d8f8e50 2024-04-25T01:35:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.501+0000 7fc043fff640 1 -- 172.21.15.16:0/1780208619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc054005220 con 0x55592d8f8e50 2024-04-25T01:35:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.501+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc05c036040 con 0x55592d8f8e50 2024-04-25T01:35:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.501+0000 7fc043fff640 1 -- 172.21.15.16:0/1780208619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc054026050 con 0x55592d8f8e50 2024-04-25T01:35:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.501+0000 7fc06dc36740 10 client.15900 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.502+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55592d90da00 con 0x55592d8f8e50 2024-04-25T01:35:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.502+0000 7fc043fff640 1 -- 172.21.15.16:0/1780208619 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc05400ee80 con 0x55592d8f8e50 2024-04-25T01:35:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.502+0000 7fc043fff640 1 -- 172.21.15.16:0/1780208619 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc05400f6a0 con 0x55592d8f8e50 2024-04-25T01:35:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.502+0000 7fc06dc36740 10 client.15900 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.502+0000 7fc06dc36740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.503+0000 7fc06dc36740 20 client.15900 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.503+0000 7fc06dc36740 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d90f120 0x55592d911590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.503+0000 7fc06dc36740 4 client.15900 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.503+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc05c036040 con 0x55592d90f120 2024-04-25T01:35:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.503+0000 7fc068cc5640 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d90f120 0x55592d911590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.504+0000 7fc068cc5640 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d90f120 0x55592d911590 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c9ba1c3c8c52230 server_cookie=2302f49122018c20 in_seq=0 out_seq=0 2024-04-25T01:35:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.504+0000 7fc043fff640 10 client.15900 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc043fff640 1 -- 172.21.15.16:0/1780208619 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc05c036040 con 0x55592d90f120 2024-04-25T01:35:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc043fff640 10 client.15900 handle_command_reply: tid=0 2024-04-25T01:35:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 client.15900 shutdown 2024-04-25T01:35:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 msgr2=0x55592d903040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d903040 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc05400af60 tx=0x7fc05400a800 comp rx=0 tx=0).stop 2024-04-25T01:35:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 shutdown_connections 2024-04-25T01:35:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8fe9b0 0x55592d90a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d90f120 0x55592d911590 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8f9840 0x55592d903580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 --2- 172.21.15.16:0/1780208619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f8e50 0x55592d903040 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 >> 172.21.15.16:0/1780208619 conn(0x55592d8f44c0 msgr2=0x55592d8f35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:47.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.505+0000 7fc043fff640 0 client.15900 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:47.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.506+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 shutdown_connections 2024-04-25T01:35:47.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.506+0000 7fc06dc36740 1 -- 172.21.15.16:0/1780208619 wait complete. 2024-04-25T01:35:47.489 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.506+0000 7fc06dc36740 20 client.15900 trim_cache size 0 max 16384 2024-04-25T01:35:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.545+0000 7fc06dc36740 1 Processor -- start 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.545+0000 7fc06dc36740 1 -- start start 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d902c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 0x55592d90a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d903190 0x55592d8f7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc06dc36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0641b1b10 con 0x55592d8fe9b0 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc06dc36740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0641af3e0 con 0x55592d903190 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc06dc36740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc0641ad230 con 0x55592d8f9840 2024-04-25T01:35:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.546+0000 7fc068cc5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 0x55592d90a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 0x55592d90a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42914/0 (socket says 172.21.15.16:42914) 2024-04-25T01:35:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 -- 172.21.15.16:0/3961318979 learned_addr learned my addr 172.21.15.16:0/3961318979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc063fff640 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d902c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc063fff640 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d903190 0x55592d8f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 -- 172.21.15.16:0/3961318979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 msgr2=0x55592d902c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d902c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 -- 172.21.15.16:0/3961318979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d903190 msgr2=0x55592d8f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d903190 0x55592d8f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.547+0000 7fc068cc5640 1 -- 172.21.15.16:0/3961318979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592d8f5230 con 0x55592d8fe9b0 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.548+0000 7fc068cc5640 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 0x55592d90a580 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x55592d8fa2b0 tx=0x7fc05c02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc7988efec40901 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.548+0000 7fc04a7fc640 1 -- 172.21.15.16:0/3961318979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c02af60 con 0x55592d8fe9b0 2024-04-25T01:35:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.548+0000 7fc04a7fc640 1 -- 172.21.15.16:0/3961318979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc05c036050 con 0x55592d8fe9b0 2024-04-25T01:35:47.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.548+0000 7fc04a7fc640 1 -- 172.21.15.16:0/3961318979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c053420 con 0x55592d8fe9b0 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.549+0000 7fc06dc36740 1 -- 172.21.15.16:0/3961318979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 msgr2=0x55592d90a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.550+0000 7fc06dc36740 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 0x55592d90a580 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x55592d8fa2b0 tx=0x7fc05c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.550+0000 7fc06dc36740 1 -- 172.21.15.16:0/3961318979 shutdown_connections 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.550+0000 7fc06dc36740 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d903190 0x55592d8f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.550+0000 7fc06dc36740 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8fe9b0 0x55592d90a580 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.550+0000 7fc06dc36740 1 --2- 172.21.15.16:0/3961318979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d8f9840 0x55592d902c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.550+0000 7fc06dc36740 1 -- 172.21.15.16:0/3961318979 >> 172.21.15.16:0/3961318979 conn(0x55592d8f44c0 msgr2=0x55592d8f48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.551+0000 7fc06dc36740 1 -- 172.21.15.16:0/3961318979 shutdown_connections 2024-04-25T01:35:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.551+0000 7fc06dc36740 1 -- 172.21.15.16:0/3961318979 wait complete. 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.551+0000 7fc06dc36740 1 -- start start 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.552+0000 7fc06dc36740 1 Processor -- start 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.552+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f9840 0x55592d907b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d908060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc06dc36740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d903190 0x55592d908fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc06dc36740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0641ad8a0 con 0x55592d8f9840 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc06dc36740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0641aed10 con 0x55592d8fe9b0 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc06dc36740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc0641adf70 con 0x55592d903190 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc06dc36740 10 client.0 _ll_register_callbacks cb 0x7fc05400aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d908060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d908060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55896/0 (socket says 172.21.15.16:55896) 2024-04-25T01:35:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 -- 172.21.15.16:0/2529261363 learned_addr learned my addr 172.21.15.16:0/2529261363 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:35:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc063fff640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d903190 0x55592d908fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.532 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc063fff640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f9840 0x55592d907b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 -- 172.21.15.16:0/2529261363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d903190 msgr2=0x55592d908fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d903190 0x55592d908fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 -- 172.21.15.16:0/2529261363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f9840 msgr2=0x55592d907b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f9840 0x55592d907b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.553+0000 7fc068cc5640 1 -- 172.21.15.16:0/2529261363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592d9095c0 con 0x55592d8fe9b0 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc068cc5640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d908060 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc05c003e90 tx=0x7fc05c003ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc0c8ce767e2386 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc043fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc043fff640 1 -- 172.21.15.16:0/2529261363 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0535d0 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc043fff640 1 -- 172.21.15.16:0/2529261363 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc05c053770 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc05c036050 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc043fff640 1 -- 172.21.15.16:0/2529261363 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc05c063050 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.554+0000 7fc06dc36740 10 client.25583 fetch_fsmap learned FSMap version 22 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55592d90a8d0 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc043fff640 1 -- 172.21.15.16:0/2529261363 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc05c036050 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc043fff640 1 -- 172.21.15.16:0/2529261363 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc05c043070 con 0x55592d8fe9b0 2024-04-25T01:35:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 10 client.25583 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 20 client.25583 populate_metadata read hostname 'smithi016' 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d9759c0 0x55592d977db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 4 client.25583 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.555+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55592d86a1f0 con 0x55592d9759c0 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.556+0000 7fc063fff640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d9759c0 0x55592d977db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.556+0000 7fc043fff640 10 client.25583 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:47.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.556+0000 7fc063fff640 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d9759c0 0x55592d977db0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37316ab5bde4b686 server_cookie=ceedbb5e779ce990 in_seq=0 out_seq=0 2024-04-25T01:35:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.579+0000 7fc043fff640 1 -- 172.21.15.16:0/2529261363 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55592d86a1f0 con 0x55592d9759c0 2024-04-25T01:35:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.579+0000 7fc043fff640 10 client.25583 handle_command_reply: tid=0 2024-04-25T01:35:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.579+0000 7fc06dc36740 1 client.25583 shutdown 2024-04-25T01:35:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.581+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 msgr2=0x55592d908060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.581+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d908060 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc05c003e90 tx=0x7fc05c003ec0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.581+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 shutdown_connections 2024-04-25T01:35:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.581+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55592d903190 0x55592d908fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.582+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55592d9759c0 0x55592d977db0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.582+0000 7fc043fff640 0 client.25583 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:35:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.582+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55592d8fe9b0 0x55592d908060 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.582+0000 7fc06dc36740 1 --2- 172.21.15.16:0/2529261363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55592d8f9840 0x55592d907b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.582+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 >> 172.21.15.16:0/2529261363 conn(0x55592d8f44c0 msgr2=0x55592d8f7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.586+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 shutdown_connections 2024-04-25T01:35:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.587+0000 7fc06dc36740 1 -- 172.21.15.16:0/2529261363 wait complete. 2024-04-25T01:35:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.587+0000 7fc06dc36740 20 client.25583 trim_cache size 0 max 16384 2024-04-25T01:35:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:35:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:35:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:35:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc04c0a2d00 msgr2=0x7fc04c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc04c0a2d00 0x7fc04c0a51c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc0580975d0 tx=0x7fc0580bb040 comp rx=0 tx=0).stop 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 msgr2=0x7fc0641a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 0x7fc0641a5480 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc05800d750 tx=0x7fc05800db50 comp rx=0 tx=0).stop 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 shutdown_connections 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc04c0a2d00 0x7fc04c0a51c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc064102250 0x7fc0641a5480 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc064101890 0x7fc06419e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 --2- 172.21.15.16:0/3805744453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0641006a0 0x7fc06419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.591+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 >> 172.21.15.16:0/3805744453 conn(0x7fc0640fbc10 msgr2=0x7fc06410d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.592+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 shutdown_connections 2024-04-25T01:35:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:35:47.592+0000 7fc06af29640 1 -- 172.21.15.16:0/3805744453 wait complete. 2024-04-25T01:35:47.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:47 smithi016 ceph-mon[29153]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 55 MiB/s wr, 937 op/s 2024-04-25T01:35:47.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:47 smithi038 ceph-mon[32841]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 55 MiB/s wr, 937 op/s 2024-04-25T01:35:47.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:47 smithi156 ceph-mon[32631]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 55 MiB/s wr, 937 op/s 2024-04-25T01:35:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:49 smithi016 ceph-mon[29153]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 64 MiB/s wr, 1.09k op/s 2024-04-25T01:35:49.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:49 smithi038 ceph-mon[32841]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 64 MiB/s wr, 1.09k op/s 2024-04-25T01:35:49.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:49 smithi156 ceph-mon[32631]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 64 MiB/s wr, 1.09k op/s 2024-04-25T01:35:51.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:51 smithi016 ceph-mon[29153]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 64 MiB/s wr, 1.09k op/s 2024-04-25T01:35:51.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:51 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:51.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:51 smithi038 ceph-mon[32841]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 64 MiB/s wr, 1.09k op/s 2024-04-25T01:35:51.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:51 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:51.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:51 smithi156 ceph-mon[32631]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 64 MiB/s wr, 1.09k op/s 2024-04-25T01:35:51.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:51 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:52.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:52.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:52.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:35:52.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:52.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:52.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:35:52.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:52.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:52.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:35:53.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:53 smithi016 ceph-mon[29153]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 61 MiB/s wr, 1.05k op/s 2024-04-25T01:35:53.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:53 smithi038 ceph-mon[32841]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 61 MiB/s wr, 1.05k op/s 2024-04-25T01:35:53.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:53 smithi156 ceph-mon[32631]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 61 MiB/s wr, 1.05k op/s 2024-04-25T01:35:55.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:55 smithi016 ceph-mon[29153]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 66 MiB/s wr, 1.13k op/s 2024-04-25T01:35:55.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:55 smithi038 ceph-mon[32841]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 66 MiB/s wr, 1.13k op/s 2024-04-25T01:35:55.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:55 smithi156 ceph-mon[32631]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 66 MiB/s wr, 1.13k op/s 2024-04-25T01:35:57.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:57 smithi016 ceph-mon[29153]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 55 MiB/s wr, 940 op/s 2024-04-25T01:35:57.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:57 smithi038 ceph-mon[32841]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 55 MiB/s wr, 940 op/s 2024-04-25T01:35:57.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:57 smithi156 ceph-mon[32631]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 55 MiB/s wr, 940 op/s 2024-04-25T01:35:59.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:35:59 smithi016 ceph-mon[29153]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 67 MiB/s wr, 1.15k op/s 2024-04-25T01:35:59.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:35:59 smithi038 ceph-mon[32841]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 67 MiB/s wr, 1.15k op/s 2024-04-25T01:35:59.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:35:59 smithi156 ceph-mon[32631]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 67 MiB/s wr, 1.15k op/s 2024-04-25T01:36:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/3477506860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700955f0 msgr2=0x7fdd70095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/3477506860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700955f0 0x7fdd70095a70 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdd78066eb0 tx=0x7fdd78096130 comp rx=0 tx=0).stop 2024-04-25T01:36:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/3477506860 shutdown_connections 2024-04-25T01:36:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/3477506860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700955f0 0x7fdd70095a70 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/3477506860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700048d0 0x7fdd70094960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/3477506860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70094400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.762+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/3477506860 >> 172.21.15.16:0/3477506860 conn(0x7fdd7008f950 msgr2=0x7fdd70091db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.763+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/3477506860 shutdown_connections 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.763+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/3477506860 wait complete. 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.763+0000 7fdd7d5f1640 1 Processor -- start 2024-04-25T01:36:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.763+0000 7fdd7d5f1640 1 -- start start 2024-04-25T01:36:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7d5f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70097380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7d5f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700048d0 0x7fdd70130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7d5f1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700955f0 0x7fdd70133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7d5f1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd700a06e0 con 0x7fdd700048d0 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7d5f1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd700a0560 con 0x7fdd70004420 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7d5f1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd700a0860 con 0x7fdd700955f0 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700048d0 0x7fdd70130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70097380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70097380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:49462/0 (socket says 172.21.15.16:49462) 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd77fff640 1 -- 172.21.15.16:0/2258419375 learned_addr learned my addr 172.21.15.16:0/2258419375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700048d0 0x7fdd70130c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33344/0 (socket says 172.21.15.16:33344) 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.764+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700955f0 0x7fdd70133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd77fff640 1 -- 172.21.15.16:0/2258419375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700955f0 msgr2=0x7fdd70133220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd77fff640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700955f0 0x7fdd70133220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd77fff640 1 -- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700048d0 msgr2=0x7fdd70130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd77fff640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700048d0 0x7fdd70130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd77fff640 1 -- 172.21.15.16:0/2258419375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd70133b30 con 0x7fdd70004420 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd77fff640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70097380 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00d770 tx=0x7fdd6c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700955f0 0x7fdd70133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd757fa640 1 -- 172.21.15.16:0/2258419375 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c017070 con 0x7fdd70004420 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd757fa640 1 -- 172.21.15.16:0/2258419375 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd6c009c30 con 0x7fdd70004420 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.765+0000 7fdd757fa640 1 -- 172.21.15.16:0/2258419375 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c003c90 con 0x7fdd70004420 2024-04-25T01:36:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.766+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7012aa30 con 0x7fdd70004420 2024-04-25T01:36:00.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.766+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd7012af30 con 0x7fdd70004420 2024-04-25T01:36:00.745 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.769+0000 7fdd81635740 1 Processor -- start 2024-04-25T01:36:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 -- start start 2024-04-25T01:36:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff5813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff5864b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff586b20 0x5560ff588fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5560ff4e4cb0 con 0x5560ff586b20 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5560ff48d5b0 con 0x5560ff5819b0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd81635740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560ff4e0c70 con 0x5560ff580fc0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff586b20 0x5560ff588fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff586b20 0x5560ff588fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33346/0 (socket says 172.21.15.16:33346) 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd7cdf0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff5813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.770+0000 7fdd777fe640 1 -- 172.21.15.16:0/696169721 learned_addr learned my addr 172.21.15.16:0/696169721 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd7cdf0640 1 -- 172.21.15.16:0/696169721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 msgr2=0x5560ff5864b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff5864b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd7cdf0640 1 -- 172.21.15.16:0/696169721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff586b20 msgr2=0x5560ff588fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff586b20 0x5560ff588fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd7cdf0640 1 -- 172.21.15.16:0/696169721 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ff1f9210 con 0x5560ff580fc0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff5813e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdd78050770 tx=0x7fdd78067890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12757c9774b7dee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd627fc640 1 -- 172.21.15.16:0/696169721 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd780ac070 con 0x5560ff580fc0 2024-04-25T01:36:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd627fc640 1 -- 172.21.15.16:0/696169721 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd7809dc00 con 0x5560ff580fc0 2024-04-25T01:36:00.749 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.771+0000 7fdd627fc640 1 -- 172.21.15.16:0/696169721 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7806a040 con 0x5560ff580fc0 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.772+0000 7fdd757fa640 1 -- 172.21.15.16:0/2258419375 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fdd6c003e30 con 0x7fdd70004420 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.772+0000 7fdd81635740 1 -- 172.21.15.16:0/696169721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 msgr2=0x5560ff5813e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.772+0000 7fdd81635740 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff5813e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdd78050770 tx=0x7fdd78067890 comp rx=0 tx=0).stop 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 -- 172.21.15.16:0/696169721 shutdown_connections 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff586b20 0x5560ff588fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff5864b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 --2- 172.21.15.16:0/696169721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff5813e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 -- 172.21.15.16:0/696169721 >> 172.21.15.16:0/696169721 conn(0x5560ff57c630 msgr2=0x5560ff57ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 -- 172.21.15.16:0/696169721 shutdown_connections 2024-04-25T01:36:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 -- 172.21.15.16:0/696169721 wait complete. 2024-04-25T01:36:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 -- start start 2024-04-25T01:36:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd757fa640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdd680a6f40 0x7fdd680a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.773+0000 7fdd81635740 1 Processor -- start 2024-04-25T01:36:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.774+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdd680a6f40 0x7fdd680a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.774+0000 7fdd757fa640 1 -- 172.21.15.16:0/2258419375 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdd6c0ce8f0 con 0x7fdd70004420 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.774+0000 7fdd81635740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff580fc0 0x5560ff58b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd81635740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff58b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff580fc0 0x5560ff58b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff580fc0 0x5560ff58b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33360/0 (socket says 172.21.15.16:33360) 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd81635740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff592960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd77fff640 1 -- 172.21.15.16:0/1220692595 learned_addr learned my addr 172.21.15.16:0/1220692595 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5560ff589a70 con 0x5560ff580fc0 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5560ff4a0b50 con 0x5560ff5819b0 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560ff589bf0 con 0x5560ff586b20 2024-04-25T01:36:00.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff58b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd81635740 10 client.0 _ll_register_callbacks cb 0x5560ff4dbca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdd680a6f40 0x7fdd680a9400 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x5560ff582210 tx=0x7fdd780649b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd777fe640 1 -- 172.21.15.16:0/1220692595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 msgr2=0x5560ff592960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff592960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd777fe640 1 -- 172.21.15.16:0/1220692595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff580fc0 msgr2=0x5560ff58b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff580fc0 0x5560ff58b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.775+0000 7fdd777fe640 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ff2c5620 con 0x5560ff5819b0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.776+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff58b970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdd640126d0 tx=0x7fdd64012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40cbe3731087a7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.776+0000 7fdd577fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.776+0000 7fdd577fe640 1 -- 172.21.15.16:0/1220692595 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6400f8c0 con 0x5560ff5819b0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.776+0000 7fdd577fe640 1 -- 172.21.15.16:0/1220692595 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdd6400fa60 con 0x5560ff5819b0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.776+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd7809dc00 con 0x5560ff5819b0 2024-04-25T01:36:00.754 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.776+0000 7fdd577fe640 1 -- 172.21.15.16:0/1220692595 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdd6400fc60 con 0x5560ff5819b0 2024-04-25T01:36:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 10 client.25592 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5560ff595df0 con 0x5560ff5819b0 2024-04-25T01:36:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd577fe640 1 -- 172.21.15.16:0/1220692595 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd7809dc00 con 0x5560ff5819b0 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd577fe640 1 -- 172.21.15.16:0/1220692595 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fdd6401c420 con 0x5560ff5819b0 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 10 client.25592 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 10 client.25592 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 20 client.25592 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5974e0 0x5560ff599910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 4 client.25592 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.777+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560ff4f21f0 con 0x5560ff5974e0 2024-04-25T01:36:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.778+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5974e0 0x5560ff599910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.778+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5974e0 0x5560ff599910 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7cc393aa255092 server_cookie=81e66d2f767ced5f in_seq=0 out_seq=0 2024-04-25T01:36:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.778+0000 7fdd577fe640 10 client.25592 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.779+0000 7fdd577fe640 1 -- 172.21.15.16:0/1220692595 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560ff4f21f0 con 0x5560ff5974e0 2024-04-25T01:36:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.779+0000 7fdd577fe640 10 client.25592 handle_command_reply: tid=0 2024-04-25T01:36:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.779+0000 7fdd81635740 1 client.25592 shutdown 2024-04-25T01:36:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.782+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 msgr2=0x5560ff58b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.782+0000 7fdd81635740 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff58b970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdd640126d0 tx=0x7fdd64012b90 comp rx=0 tx=0).stop 2024-04-25T01:36:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 shutdown_connections 2024-04-25T01:36:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd81635740 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff592960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd81635740 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5974e0 0x5560ff599910 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd81635740 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff5819b0 0x5560ff58b970 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd81635740 1 --2- 172.21.15.16:0/1220692595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff580fc0 0x5560ff58b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 >> 172.21.15.16:0/1220692595 conn(0x5560ff57c630 msgr2=0x5560ff57b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.786+0000 7fdd577fe640 0 client.25592 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.787+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 shutdown_connections 2024-04-25T01:36:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.787+0000 7fdd81635740 1 -- 172.21.15.16:0/1220692595 wait complete. 2024-04-25T01:36:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.787+0000 7fdd81635740 20 client.25592 trim_cache size 0 max 16384 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 Processor -- start 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 -- start start 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff582e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff592920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff586b20 0x5560ff583390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7013d820 con 0x5560ff5819b0 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7013b0f0 con 0x5560ff586b20 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd81635740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd70138f40 con 0x5560ff580fc0 2024-04-25T01:36:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.814+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff582e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff582e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:58742/0 (socket says 172.21.15.16:58742) 2024-04-25T01:36:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 -- 172.21.15.16:0/1188741548 learned_addr learned my addr 172.21.15.16:0/1188741548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff586b20 0x5560ff583390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff592920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 -- 172.21.15.16:0/1188741548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff586b20 msgr2=0x5560ff583390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff586b20 0x5560ff583390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 -- 172.21.15.16:0/1188741548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 msgr2=0x5560ff592920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff592920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 -- 172.21.15.16:0/1188741548 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ff595c10 con 0x5560ff580fc0 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff582e50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009610 tx=0x7fdd6c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbe3e05bfbea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd61ffb640 1 -- 172.21.15.16:0/1188741548 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0b3030 con 0x5560ff580fc0 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd61ffb640 1 -- 172.21.15.16:0/1188741548 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd6c015020 con 0x5560ff580fc0 2024-04-25T01:36:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.815+0000 7fdd61ffb640 1 -- 172.21.15.16:0/1188741548 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c020070 con 0x5560ff580fc0 2024-04-25T01:36:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 -- 172.21.15.16:0/1188741548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 msgr2=0x5560ff582e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff582e50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009610 tx=0x7fdd6c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 -- 172.21.15.16:0/1188741548 shutdown_connections 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff586b20 0x5560ff583390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff592920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 --2- 172.21.15.16:0/1188741548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff580fc0 0x5560ff582e50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 -- 172.21.15.16:0/1188741548 >> 172.21.15.16:0/1188741548 conn(0x5560ff57c630 msgr2=0x5560ff58b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 -- 172.21.15.16:0/1188741548 shutdown_connections 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 -- 172.21.15.16:0/1188741548 wait complete. 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.817+0000 7fdd81635740 1 -- start start 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 Processor -- start 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff580fc0 0x5560ff58fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff5902d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff5912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd701395b0 con 0x5560ff5819b0 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7013aa20 con 0x5560ff580fc0 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd70139c80 con 0x5560ff586b20 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd81635740 10 client.0 _ll_register_callbacks cb 0x7fdd64009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff580fc0 0x5560ff58fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff5902d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff580fc0 0x5560ff58fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:49494/0 (socket says 172.21.15.16:49494) 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd77fff640 1 -- 172.21.15.16:0/1728054428 learned_addr learned my addr 172.21.15.16:0/1728054428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.818+0000 7fdd7cdf0640 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff5912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd777fe640 1 -- 172.21.15.16:0/1728054428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 msgr2=0x5560ff5912e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff5912e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd777fe640 1 -- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff580fc0 msgr2=0x5560ff58fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff580fc0 0x5560ff58fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd777fe640 1 -- 172.21.15.16:0/1728054428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ff5918d0 con 0x5560ff5819b0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd777fe640 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff5902d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdd6401ca30 tx=0x7fdd6400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4490026f831ccfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd577fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd577fe640 1 -- 172.21.15.16:0/1728054428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6401ccc0 con 0x5560ff5819b0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd6c015020 con 0x5560ff5819b0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd577fe640 1 -- 172.21.15.16:0/1728054428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdd640050b0 con 0x5560ff5819b0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd577fe640 1 -- 172.21.15.16:0/1728054428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdd64036050 con 0x5560ff5819b0 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd81635740 10 client.15924 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.819+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5560ff592c50 con 0x5560ff5819b0 2024-04-25T01:36:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd577fe640 1 -- 172.21.15.16:0/1728054428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd6c015020 con 0x5560ff5819b0 2024-04-25T01:36:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd577fe640 1 -- 172.21.15.16:0/1728054428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fdd64003b50 con 0x5560ff5819b0 2024-04-25T01:36:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd81635740 10 client.15924 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd81635740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd81635740 20 client.15924 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd81635740 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5fe2c0 0x5560ff6006e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd81635740 4 client.15924 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5560ff4ec540 con 0x5560ff5fe2c0 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.820+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5fe2c0 0x5560ff6006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.821+0000 7fdd577fe640 10 client.15924 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.821+0000 7fdd77fff640 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5fe2c0 0x5560ff6006e0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f249efbddf115196 server_cookie=c21716ab6b496601 in_seq=0 out_seq=0 2024-04-25T01:36:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.827+0000 7fdd577fe640 1 -- 172.21.15.16:0/1728054428 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5560ff4ec540 con 0x5560ff5fe2c0 2024-04-25T01:36:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.827+0000 7fdd577fe640 10 client.15924 handle_command_reply: tid=0 2024-04-25T01:36:00.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.827+0000 7fdd81635740 1 client.15924 shutdown 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.827+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 msgr2=0x5560ff5902d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.828+0000 7fdd81635740 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff5902d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdd6401ca30 tx=0x7fdd6400fa40 comp rx=0 tx=0).stop 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.828+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 shutdown_connections 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.828+0000 7fdd81635740 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ff586b20 0x5560ff5912e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.828+0000 7fdd81635740 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5560ff5819b0 0x5560ff5902d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd81635740 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5560ff5fe2c0 0x5560ff6006e0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd81635740 1 --2- 172.21.15.16:0/1728054428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560ff580fc0 0x5560ff58fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd577fe640 0 client.15924 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 >> 172.21.15.16:0/1728054428 conn(0x5560ff57c630 msgr2=0x5560ff58b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 shutdown_connections 2024-04-25T01:36:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd81635740 1 -- 172.21.15.16:0/1728054428 wait complete. 2024-04-25T01:36:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.829+0000 7fdd81635740 20 client.15924 trim_cache size 0 max 16384 2024-04-25T01:36:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:36:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:36:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:36:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.830+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdd680a6f40 msgr2=0x7fdd680a9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.830+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdd680a6f40 0x7fdd680a9400 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x5560ff582210 tx=0x7fdd780649b0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.830+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 msgr2=0x7fdd70097380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.830+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70097380 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00d770 tx=0x7fdd6c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 shutdown_connections 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fdd680a6f40 0x7fdd680a9400 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd700955f0 0x7fdd70133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd700048d0 0x7fdd70130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 --2- 172.21.15.16:0/2258419375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd70004420 0x7fdd70097380 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 >> 172.21.15.16:0/2258419375 conn(0x7fdd7008f950 msgr2=0x7fdd70091120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 shutdown_connections 2024-04-25T01:36:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:00.831+0000 7fdd7d5f1640 1 -- 172.21.15.16:0/2258419375 wait complete. 2024-04-25T01:36:01.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:01 smithi016 ceph-mon[29153]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 63 MiB/s wr, 1.09k op/s 2024-04-25T01:36:01.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:01 smithi038 ceph-mon[32841]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 63 MiB/s wr, 1.09k op/s 2024-04-25T01:36:01.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:01 smithi156 ceph-mon[32631]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 63 MiB/s wr, 1.09k op/s 2024-04-25T01:36:03.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:03 smithi016 ceph-mon[29153]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 60 MiB/s wr, 1.04k op/s 2024-04-25T01:36:03.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:03 smithi038 ceph-mon[32841]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 60 MiB/s wr, 1.04k op/s 2024-04-25T01:36:03.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:03 smithi156 ceph-mon[32631]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 60 MiB/s wr, 1.04k op/s 2024-04-25T01:36:05.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:05 smithi016 ceph-mon[29153]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 65 MiB/s wr, 1.13k op/s 2024-04-25T01:36:05.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:05 smithi038 ceph-mon[32841]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 65 MiB/s wr, 1.13k op/s 2024-04-25T01:36:05.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:05 smithi156 ceph-mon[32631]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 65 MiB/s wr, 1.13k op/s 2024-04-25T01:36:07.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:07 smithi016 ceph-mon[29153]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 53 MiB/s wr, 938 op/s 2024-04-25T01:36:07.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:07 smithi038 ceph-mon[32841]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 53 MiB/s wr, 938 op/s 2024-04-25T01:36:07.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:07 smithi156 ceph-mon[32631]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 53 MiB/s wr, 938 op/s 2024-04-25T01:36:09.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:09 smithi016 ceph-mon[29153]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:09.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:09 smithi038 ceph-mon[32841]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:09.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:09 smithi156 ceph-mon[32631]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:11.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:11 smithi016 ceph-mon[29153]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:36:11.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:11 smithi038 ceph-mon[32841]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:36:11.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:11 smithi156 ceph-mon[32631]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:36:13.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:13 smithi038 ceph-mon[32841]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 59 MiB/s wr, 1.05k op/s 2024-04-25T01:36:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 -- 172.21.15.16:0/1924442192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a0107c00 msgr2=0x7f09a0108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 --2- 172.21.15.16:0/1924442192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a0107c00 0x7f09a0108020 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0998009ca0 tx=0x7f099802a8b0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 -- 172.21.15.16:0/1924442192 shutdown_connections 2024-04-25T01:36:13.965 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 --2- 172.21.15.16:0/1924442192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a01097b0 0x7f09a0110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 --2- 172.21.15.16:0/1924442192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a0108df0 0x7f09a0109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 --2- 172.21.15.16:0/1924442192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a0107c00 0x7f09a0108020 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.988+0000 7f09a6093640 1 -- 172.21.15.16:0/1924442192 >> 172.21.15.16:0/1924442192 conn(0x7f09a0076b80 msgr2=0x7f09a0078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.989+0000 7f09a6093640 1 -- 172.21.15.16:0/1924442192 shutdown_connections 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.989+0000 7f09a6093640 1 -- 172.21.15.16:0/1924442192 wait complete. 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.989+0000 7f09a6093640 1 Processor -- start 2024-04-25T01:36:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 -- start start 2024-04-25T01:36:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a0107c00 0x7f09a019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a0108df0 0x7f09a019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 0x7f09a01a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a0113750 con 0x7f09a0107c00 2024-04-25T01:36:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01135d0 con 0x7f09a0108df0 2024-04-25T01:36:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f09a6093640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01138d0 con 0x7f09a01097b0 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f099f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a0107c00 0x7f09a019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f099ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 0x7f09a01a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f099ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 0x7f09a01a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:33940/0 (socket says 172.21.15.16:33940) 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f099ffff640 1 -- 172.21.15.16:0/2716695736 learned_addr learned my addr 172.21.15.16:0/2716695736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.990+0000 7f099effd640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a0108df0 0x7f09a019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099ffff640 1 -- 172.21.15.16:0/2716695736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a0108df0 msgr2=0x7f09a019e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099ffff640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a0108df0 0x7f09a019e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099ffff640 1 -- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a0107c00 msgr2=0x7f09a019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099ffff640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a0107c00 0x7f09a019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099ffff640 1 -- 172.21.15.16:0/2716695736 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a01a5a60 con 0x7f09a01097b0 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099f7fe640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a0107c00 0x7f09a019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.991+0000 7f099ffff640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 0x7f09a01a53f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f099000e970 tx=0x7f099000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.992+0000 7f099cff9640 1 -- 172.21.15.16:0/2716695736 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099000ccc0 con 0x7f09a01097b0 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.992+0000 7f099cff9640 1 -- 172.21.15.16:0/2716695736 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f099000ce60 con 0x7f09a01097b0 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.992+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a01a5d50 con 0x7f09a01097b0 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.992+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a01a6240 con 0x7f09a01097b0 2024-04-25T01:36:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.992+0000 7f099cff9640 1 -- 172.21.15.16:0/2716695736 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990010430 con 0x7f09a01097b0 2024-04-25T01:36:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.996+0000 7f09a8d9d740 1 Processor -- start 2024-04-25T01:36:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.996+0000 7f099cff9640 1 -- 172.21.15.16:0/2716695736 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0990004fd0 con 0x7f09a01097b0 2024-04-25T01:36:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.996+0000 7f09a8d9d740 1 -- start start 2024-04-25T01:36:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801030 0x560076801450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801a20 0x560076806520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076809020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560076764cb0 con 0x560076801030 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56007670d5b0 con 0x560076801a20 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560076760c70 con 0x560076806b90 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f099effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801030 0x560076801450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f099effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801030 0x560076801450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40908/0 (socket says 172.21.15.16:40908) 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f099effd640 1 -- 172.21.15.16:0/2248415215 learned_addr learned my addr 172.21.15.16:0/2248415215 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f099effd640 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076809020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.997+0000 7f099f7fe640 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801a20 0x560076806520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099cff9640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09880a2c00 0x7f09880a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099ffff640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09880a2c00 0x7f09880a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099f7fe640 1 -- 172.21.15.16:0/2248415215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 msgr2=0x560076809020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099f7fe640 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076809020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099f7fe640 1 -- 172.21.15.16:0/2248415215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801030 msgr2=0x560076801450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099f7fe640 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801030 0x560076801450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099f7fe640 1 -- 172.21.15.16:0/2248415215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560076479210 con 0x560076801a20 2024-04-25T01:36:13.975 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099cff9640 1 -- 172.21.15.16:0/2716695736 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f09900cf3f0 con 0x7f09a01097b0 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.998+0000 7f099f7fe640 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801a20 0x560076806520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f099802aea0 tx=0x7f0998030f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2578b8911b5836a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.999+0000 7f099ffff640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09880a2c00 0x7f09880a50c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0990098600 tx=0x7f09900c67c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.999+0000 7f0985ffb640 1 -- 172.21.15.16:0/2248415215 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998030b60 con 0x560076801a20 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.999+0000 7f0985ffb640 1 -- 172.21.15.16:0/2248415215 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0998036030 con 0x560076801a20 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:13.999+0000 7f0985ffb640 1 -- 172.21.15.16:0/2248415215 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998004790 con 0x560076801a20 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2248415215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801a20 msgr2=0x560076806520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801a20 0x560076806520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f099802aea0 tx=0x7f0998030f60 comp rx=0 tx=0).stop 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2248415215 shutdown_connections 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076809020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801a20 0x560076806520 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2248415215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801030 0x560076801450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2248415215 >> 172.21.15.16:0/2248415215 conn(0x5600767fc6a0 msgr2=0x5600767fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2248415215 shutdown_connections 2024-04-25T01:36:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.000+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2248415215 wait complete. 2024-04-25T01:36:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 -- start start 2024-04-25T01:36:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 Processor -- start 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x56007680ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x560076812a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560076809ae0 con 0x560076801a20 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560076720b50 con 0x560076806b90 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.001+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560076809c60 con 0x560076801030 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:33954/0 (socket says 172.21.15.16:33954) 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 -- 172.21.15.16:0/3487964578 learned_addr learned my addr 172.21.15.16:0/3487964578 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f09a8d9d740 10 client.0 _ll_register_callbacks cb 0x56007675bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x560076812a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099f7fe640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x56007680ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 -- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 msgr2=0x560076812a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x560076812a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 -- 172.21.15.16:0/3487964578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 msgr2=0x56007680ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x56007680ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 -- 172.21.15.16:0/3487964578 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560076545620 con 0x560076801030 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f099effd640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b4c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0994013d20 tx=0x7f0994005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d33678e36d9ec567 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.002+0000 7f097effd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f097effd640 1 -- 172.21.15.16:0/3487964578 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099400fe60 con 0x560076801030 2024-04-25T01:36:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0998036030 con 0x560076801030 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f097effd640 1 -- 172.21.15.16:0/3487964578 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0994010430 con 0x560076801030 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f097effd640 1 -- 172.21.15.16:0/3487964578 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0994010630 con 0x560076801030 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f09a8d9d740 10 client.25564 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560076815e50 con 0x560076801030 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.003+0000 7f097effd640 1 -- 172.21.15.16:0/3487964578 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0998036030 con 0x560076801030 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f097effd640 1 -- 172.21.15.16:0/3487964578 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f099402d720 con 0x560076801030 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f09a8d9d740 10 client.25564 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f09a8d9d740 10 client.25564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f09a8d9d740 20 client.25564 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5600768174d0 0x560076819960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f09a8d9d740 4 client.25564 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600767721f0 con 0x5600768174d0 2024-04-25T01:36:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.004+0000 7f099f7fe640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5600768174d0 0x560076819960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.005+0000 7f097effd640 10 client.25564 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.005+0000 7f099f7fe640 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5600768174d0 0x560076819960 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=271cf0239d40402e server_cookie=13484f4f137caf44 in_seq=0 out_seq=0 2024-04-25T01:36:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f097effd640 1 -- 172.21.15.16:0/3487964578 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600767721f0 con 0x5600768174d0 2024-04-25T01:36:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f097effd640 10 client.25564 handle_command_reply: tid=0 2024-04-25T01:36:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 client.25564 shutdown 2024-04-25T01:36:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 msgr2=0x56007680b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b4c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0994013d20 tx=0x7f0994005f90 comp rx=0 tx=0).stop 2024-04-25T01:36:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 shutdown_connections 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5600768174d0 0x560076819960 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f097effd640 0 client.25564 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x560076812a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x56007680ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/3487964578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b4c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.006+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 >> 172.21.15.16:0/3487964578 conn(0x5600767fc6a0 msgr2=0x5600767fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.007+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 shutdown_connections 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.007+0000 7f09a8d9d740 1 -- 172.21.15.16:0/3487964578 wait complete. 2024-04-25T01:36:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.007+0000 7f09a8d9d740 20 client.25564 trim_cache size 0 max 16384 2024-04-25T01:36:13.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:13 smithi156 ceph-mon[32631]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 59 MiB/s wr, 1.05k op/s 2024-04-25T01:36:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.036+0000 7f09a8d9d740 1 Processor -- start 2024-04-25T01:36:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.036+0000 7f09a8d9d740 1 -- start start 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x560076812a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x5600767ff650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01b18c0 con 0x560076801a20 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01af190 con 0x560076806b90 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01acfe0 con 0x560076801030 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f099effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f099effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:33968/0 (socket says 172.21.15.16:33968) 2024-04-25T01:36:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f099effd640 1 -- 172.21.15.16:0/2470518306 learned_addr learned my addr 172.21.15.16:0/2470518306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f099f7fe640 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x560076812a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.037+0000 7f099effd640 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x5600767ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f099effd640 1 -- 172.21.15.16:0/2470518306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 msgr2=0x5600767ff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f099effd640 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x5600767ff650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f099effd640 1 -- 172.21.15.16:0/2470518306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 msgr2=0x560076812a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f099effd640 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x560076812a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f099effd640 1 -- 172.21.15.16:0/2470518306 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600768061d0 con 0x560076801030 2024-04-25T01:36:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f099effd640 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0994009f70 tx=0x7f0994009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=369fb4fb83c7be87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f09857fa640 1 -- 172.21.15.16:0/2470518306 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099400fe60 con 0x560076801030 2024-04-25T01:36:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f09857fa640 1 -- 172.21.15.16:0/2470518306 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0994010430 con 0x560076801030 2024-04-25T01:36:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.038+0000 7f09857fa640 1 -- 172.21.15.16:0/2470518306 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994005290 con 0x560076801030 2024-04-25T01:36:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.040+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2470518306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 msgr2=0x56007680b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:14.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.040+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0994009f70 tx=0x7f0994009f00 comp rx=0 tx=0).stop 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.040+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2470518306 shutdown_connections 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.040+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076806b90 0x5600767ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.041+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x560076812a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.041+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/2470518306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076801030 0x56007680b960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.041+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2470518306 >> 172.21.15.16:0/2470518306 conn(0x5600767fc6a0 msgr2=0x5600767fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.042+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2470518306 shutdown_connections 2024-04-25T01:36:14.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.042+0000 7f09a8d9d740 1 -- 172.21.15.16:0/2470518306 wait complete. 2024-04-25T01:36:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 -- start start 2024-04-25T01:36:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 Processor -- start 2024-04-25T01:36:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801030 0x56007680ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x5600768104f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076811500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01ad650 con 0x560076801a20 2024-04-25T01:36:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01aeac0 con 0x560076801030 2024-04-25T01:36:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09a01add20 con 0x560076806b90 2024-04-25T01:36:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.043+0000 7f09a8d9d740 10 client.0 _ll_register_callbacks cb 0x7f0994013680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x5600768104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x5600768104f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40942/0 (socket says 172.21.15.16:40942) 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 -- 172.21.15.16:0/557868702 learned_addr learned my addr 172.21.15.16:0/557868702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099effd640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076811500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099effd640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801030 0x56007680ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 -- 172.21.15.16:0/557868702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 msgr2=0x560076811500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076811500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 -- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801030 msgr2=0x56007680ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801030 0x56007680ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 -- 172.21.15.16:0/557868702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560076811af0 con 0x560076801a20 2024-04-25T01:36:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.044+0000 7f099f7fe640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x5600768104f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f099802aea0 tx=0x7f09980095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce0cdc662486e27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.045+0000 7f097effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.045+0000 7f097effd640 1 -- 172.21.15.16:0/557868702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998030960 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.045+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0994010430 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.045+0000 7f097effd640 1 -- 172.21.15.16:0/557868702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0998030b00 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.046+0000 7f097effd640 1 -- 172.21.15.16:0/557868702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0998030d00 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.046+0000 7f09a8d9d740 10 client.15942 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.046+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56007687dae0 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.046+0000 7f097effd640 1 -- 172.21.15.16:0/557868702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0994010430 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f097effd640 1 -- 172.21.15.16:0/557868702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0998043070 con 0x560076801a20 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f09a8d9d740 10 client.15942 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f09a8d9d740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f09a8d9d740 20 client.15942 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56007687df30 0x560076880320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f09a8d9d740 4 client.15942 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:36:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56007676c540 con 0x56007687df30 2024-04-25T01:36:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f099effd640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56007687df30 0x560076880320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f097effd640 10 client.15942 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.047+0000 7f099effd640 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56007687df30 0x560076880320 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d42f34e47db9a1b0 server_cookie=7dfb0b46e15860fe in_seq=0 out_seq=0 2024-04-25T01:36:14.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f097effd640 1 -- 172.21.15.16:0/557868702 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56007676c540 con 0x56007687df30 2024-04-25T01:36:14.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f097effd640 10 client.15942 handle_command_reply: tid=0 2024-04-25T01:36:14.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 client.15942 shutdown 2024-04-25T01:36:14.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 msgr2=0x5600768104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x5600768104f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f099802aea0 tx=0x7f09980095c0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 shutdown_connections 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560076806b90 0x560076811500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560076801a20 0x5600768104f0 secure :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f099802aea0 tx=0x7f09980095c0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x56007687df30 0x560076880320 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.052+0000 7f09a8d9d740 1 --2- 172.21.15.16:0/557868702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560076801030 0x56007680ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.053+0000 7f097effd640 0 client.15942 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.053+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 >> 172.21.15.16:0/557868702 conn(0x5600767fc6a0 msgr2=0x5600767ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.053+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 shutdown_connections 2024-04-25T01:36:14.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.053+0000 7f09a8d9d740 1 -- 172.21.15.16:0/557868702 wait complete. 2024-04-25T01:36:14.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.053+0000 7f09a8d9d740 20 client.15942 trim_cache size 0 max 16384 2024-04-25T01:36:14.030 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:36:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:36:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:36:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.056+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09880a2c00 msgr2=0x7f09880a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.056+0000 7f09a6093640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09880a2c00 0x7f09880a50c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0990098600 tx=0x7f09900c67c0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.056+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 msgr2=0x7f09a01a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.056+0000 7f09a6093640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 0x7f09a01a53f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f099000e970 tx=0x7f099000ed70 comp rx=0 tx=0).stop 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 shutdown_connections 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f09880a2c00 0x7f09880a50c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a01097b0 0x7f09a01a53f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09a0108df0 0x7f09a019e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 --2- 172.21.15.16:0/2716695736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09a0107c00 0x7f09a019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 >> 172.21.15.16:0/2716695736 conn(0x7f09a0076b80 msgr2=0x7f09a010c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 shutdown_connections 2024-04-25T01:36:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:14.057+0000 7f09a6093640 1 -- 172.21.15.16:0/2716695736 wait complete. 2024-04-25T01:36:14.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:13 smithi016 ceph-mon[29153]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 59 MiB/s wr, 1.05k op/s 2024-04-25T01:36:14.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:14 smithi038 ceph-mon[32841]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:14.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:14 smithi156 ceph-mon[32631]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:15.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:14 smithi016 ceph-mon[29153]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:17.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:16 smithi016 ceph-mon[29153]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 54 MiB/s wr, 959 op/s 2024-04-25T01:36:17.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:16 smithi038 ceph-mon[32841]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 54 MiB/s wr, 959 op/s 2024-04-25T01:36:17.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:16 smithi156 ceph-mon[32631]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 54 MiB/s wr, 959 op/s 2024-04-25T01:36:19.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:18 smithi016 ceph-mon[29153]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 66 MiB/s wr, 1.17k op/s 2024-04-25T01:36:19.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:18 smithi038 ceph-mon[32841]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 66 MiB/s wr, 1.17k op/s 2024-04-25T01:36:19.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:18 smithi156 ceph-mon[32631]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 66 MiB/s wr, 1.17k op/s 2024-04-25T01:36:21.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:20 smithi016 ceph-mon[29153]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 60 MiB/s wr, 1.05k op/s 2024-04-25T01:36:21.214 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:20 smithi038 ceph-mon[32841]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 60 MiB/s wr, 1.05k op/s 2024-04-25T01:36:21.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:20 smithi156 ceph-mon[32631]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 60 MiB/s wr, 1.05k op/s 2024-04-25T01:36:23.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:22 smithi016 ceph-mon[29153]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 61 MiB/s wr, 1.07k op/s 2024-04-25T01:36:23.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:22 smithi038 ceph-mon[32841]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 61 MiB/s wr, 1.07k op/s 2024-04-25T01:36:23.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:22 smithi156 ceph-mon[32631]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 61 MiB/s wr, 1.07k op/s 2024-04-25T01:36:25.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:24 smithi016 ceph-mon[29153]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 66 MiB/s wr, 1.16k op/s 2024-04-25T01:36:25.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:24 smithi038 ceph-mon[32841]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 66 MiB/s wr, 1.16k op/s 2024-04-25T01:36:25.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:24 smithi156 ceph-mon[32631]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 66 MiB/s wr, 1.16k op/s 2024-04-25T01:36:27.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:26 smithi016 ceph-mon[29153]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 55 MiB/s wr, 963 op/s 2024-04-25T01:36:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.208+0000 7f1c13e72640 1 -- 172.21.15.16:0/900099505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 msgr2=0x7f1c0c10d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.208+0000 7f1c13e72640 1 --2- 172.21.15.16:0/900099505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c10d520 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009900 tx=0x7f1bfc02a440 comp rx=0 tx=0).stop 2024-04-25T01:36:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 -- 172.21.15.16:0/900099505 shutdown_connections 2024-04-25T01:36:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 --2- 172.21.15.16:0/900099505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c10da60 0x7f1c0c10dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 --2- 172.21.15.16:0/900099505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c10d520 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 --2- 172.21.15.16:0/900099505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c0c108740 0x7f1c0c10ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 -- 172.21.15.16:0/900099505 >> 172.21.15.16:0/900099505 conn(0x7f1c0c06e7b0 msgr2=0x7f1c0c070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 -- 172.21.15.16:0/900099505 shutdown_connections 2024-04-25T01:36:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.210+0000 7f1c13e72640 1 -- 172.21.15.16:0/900099505 wait complete. 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 Processor -- start 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 -- start start 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 0x7f1c0c117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c118150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c0c10da60 0x7f1c0c119190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c110340 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1101c0 con 0x7f1c0c108740 2024-04-25T01:36:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.211+0000 7f1c13e72640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1104c0 con 0x7f1c0c10da60 2024-04-25T01:36:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.212+0000 7f1c11c0e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 0x7f1c0c117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.212+0000 7f1c11c0e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 0x7f1c0c117c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60364/0 (socket says 172.21.15.16:60364) 2024-04-25T01:36:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.212+0000 7f1c11c0e640 1 -- 172.21.15.16:0/3709798629 learned_addr learned my addr 172.21.15.16:0/3709798629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.212+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c118150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c1140d640 1 -- 172.21.15.16:0/3709798629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c0c10da60 msgr2=0x7f1c0c119190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c1240f640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c0c10da60 0x7f1c0c119190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c0c10da60 0x7f1c0c119190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c1140d640 1 -- 172.21.15.16:0/3709798629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 msgr2=0x7f1c0c117c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 0x7f1c0c117c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c1140d640 1 -- 172.21.15.16:0/3709798629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c119800 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.213+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 0x7f1c0c117c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.214+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c118150 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc02aa30 tx=0x7f1bfc030b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.216+0000 7f1c02ffd640 1 -- 172.21.15.16:0/3709798629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc040020 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.217+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c119a90 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.217+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c11a000 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.217+0000 7f1c02ffd640 1 -- 172.21.15.16:0/3709798629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bfc004250 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.217+0000 7f1c02ffd640 1 -- 172.21.15.16:0/3709798629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc035070 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.220+0000 7f1c16b80740 1 Processor -- start 2024-04-25T01:36:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 -- start start 2024-04-25T01:36:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fa8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fad340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fafe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5596c1f0bcb0 con 0x5596c1fad9b0 2024-04-25T01:36:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5596c1eb45b0 con 0x5596c1fa8840 2024-04-25T01:36:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.222+0000 7f1c16b80740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5596c1f07c70 con 0x5596c1fa7e50 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fa8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fa8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:52290/0 (socket says 172.21.15.16:52290) 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 -- 172.21.15.16:0/2021715095 learned_addr learned my addr 172.21.15.16:0/2021715095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 -- 172.21.15.16:0/2021715095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 msgr2=0x5596c1fad340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fad340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 -- 172.21.15.16:0/2021715095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 msgr2=0x5596c1fafe40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fafe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.223+0000 7f1c1240f640 1 -- 172.21.15.16:0/2021715095 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596c1c20210 con 0x5596c1fa7e50 2024-04-25T01:36:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.224+0000 7f1c1240f640 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fa8270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c04007d90 tx=0x7f1c0400c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2021428bceab4314 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.224+0000 7f1c02ffd640 1 -- 172.21.15.16:0/3709798629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1bfc043020 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.224+0000 7f1bf7fff640 1 -- 172.21.15.16:0/2021715095 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04015020 con 0x5596c1fa7e50 2024-04-25T01:36:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.224+0000 7f1bf7fff640 1 -- 172.21.15.16:0/2021715095 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c04011ae0 con 0x5596c1fa7e50 2024-04-25T01:36:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.225+0000 7f1c02ffd640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bf80a2de0 0x7f1bf80a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.225+0000 7f1c16b80740 1 -- 172.21.15.16:0/2021715095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 msgr2=0x5596c1fa8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.225+0000 7f1c16b80740 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fa8270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c04007d90 tx=0x7f1c0400c560 comp rx=0 tx=0).stop 2024-04-25T01:36:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.225+0000 7f1c02ffd640 1 -- 172.21.15.16:0/3709798629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1bfc0effd0 con 0x7f1c0c10b0b0 2024-04-25T01:36:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.225+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bf80a2de0 0x7f1bf80a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.226+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bf80a2de0 0x7f1bf80a52a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c08006fd0 tx=0x7f1c08008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.228+0000 7f1c16b80740 1 -- 172.21.15.16:0/2021715095 shutdown_connections 2024-04-25T01:36:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.228+0000 7f1c16b80740 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fafe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.228+0000 7f1c16b80740 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.228+0000 7f1c16b80740 1 --2- 172.21.15.16:0/2021715095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fa8270 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.228+0000 7f1c16b80740 1 -- 172.21.15.16:0/2021715095 >> 172.21.15.16:0/2021715095 conn(0x5596c1fa34c0 msgr2=0x5596c1fa38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.229+0000 7f1c16b80740 1 -- 172.21.15.16:0/2021715095 shutdown_connections 2024-04-25T01:36:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.229+0000 7f1c16b80740 1 -- 172.21.15.16:0/2021715095 wait complete. 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.229+0000 7f1c16b80740 1 -- start start 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.229+0000 7f1c16b80740 1 Processor -- start 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa8840 0x5596c1fb28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 0x5596c1fb9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5596c1fb09b0 con 0x5596c1fa8840 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5596c1ec7b50 con 0x5596c1fad9b0 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5596c1fb0b30 con 0x5596c1fa7e50 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c16b80740 10 client.0 _ll_register_callbacks cb 0x5596c1f02ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:52296/0 (socket says 172.21.15.16:52296) 2024-04-25T01:36:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 -- 172.21.15.16:0/3027860421 learned_addr learned my addr 172.21.15.16:0/3027860421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa8840 0x5596c1fb28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 -- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 msgr2=0x5596c1fb9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 0x5596c1fb9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 -- 172.21.15.16:0/3027860421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa8840 msgr2=0x5596c1fb28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa8840 0x5596c1fb28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.230+0000 7f1c11c0e640 1 -- 172.21.15.16:0/3027860421 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596c1cec620 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800de70 tx=0x7f1c08004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9009b082985a973 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1bf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1bf57fa640 1 -- 172.21.15.16:0/3027860421 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08024060 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1bf57fa640 1 -- 172.21.15.16:0/3027860421 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1c0801f400 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c04011ae0 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1bf57fa640 1 -- 172.21.15.16:0/3027860421 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1c08005970 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1c16b80740 10 client.25576 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.231+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5596c1fbcd40 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1bf57fa640 1 -- 172.21.15.16:0/3027860421 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c04011ae0 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1bf57fa640 1 -- 172.21.15.16:0/3027860421 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1c08013070 con 0x5596c1fa7e50 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c16b80740 10 client.25576 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c16b80740 10 client.25576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c16b80740 20 client.25576 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c16b80740 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c1fbe490 0x5596c1fc0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c16b80740 4 client.25576 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596c1f191f0 con 0x5596c1fbe490 2024-04-25T01:36:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.232+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c1fbe490 0x5596c1fc0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.235+0000 7f1bf57fa640 10 client.25576 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.235+0000 7f1c1140d640 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c1fbe490 0x5596c1fc0930 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5a50761747dce32 server_cookie=761d82cabe777115 in_seq=0 out_seq=0 2024-04-25T01:36:27.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:26 smithi038 ceph-mon[32841]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 55 MiB/s wr, 963 op/s 2024-04-25T01:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.236+0000 7f1bf57fa640 1 -- 172.21.15.16:0/3027860421 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596c1f191f0 con 0x5596c1fbe490 2024-04-25T01:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.237+0000 7f1bf57fa640 10 client.25576 handle_command_reply: tid=0 2024-04-25T01:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.237+0000 7f1c16b80740 1 client.25576 shutdown 2024-04-25T01:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.237+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 msgr2=0x5596c1fb2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.237+0000 7f1c16b80740 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800de70 tx=0x7f1c08004750 comp rx=0 tx=0).stop 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 shutdown_connections 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c1fbe490 0x5596c1fc0930 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 0x5596c1fb9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa8840 0x5596c1fb28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 --2- 172.21.15.16:0/3027860421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2380 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 >> 172.21.15.16:0/3027860421 conn(0x5596c1fa34c0 msgr2=0x5596c1fa25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1bf57fa640 0 client.25576 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 shutdown_connections 2024-04-25T01:36:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 1 -- 172.21.15.16:0/3027860421 wait complete. 2024-04-25T01:36:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.239+0000 7f1c16b80740 20 client.25576 trim_cache size 0 max 16384 2024-04-25T01:36:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.267+0000 7f1c16b80740 1 Processor -- start 2024-04-25T01:36:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.267+0000 7f1c16b80740 1 -- start start 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.267+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.267+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fb98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c11c0e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c11c0e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:52308/0 (socket says 172.21.15.16:52308) 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c11c0e640 1 -- 172.21.15.16:0/4088873483 learned_addr learned my addr 172.21.15.16:0/4088873483 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c1140d640 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fb98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c16b80740 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fa6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf0010f50 con 0x5596c1fad9b0 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf000e820 con 0x5596c1fa8840 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf000c670 con 0x5596c1fa7e50 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.268+0000 7f1c1240f640 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fa6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1c11c0e640 1 -- 172.21.15.16:0/4088873483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 msgr2=0x5596c1fb98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fb98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1c11c0e640 1 -- 172.21.15.16:0/4088873483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 msgr2=0x5596c1fa6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fa6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1c11c0e640 1 -- 172.21.15.16:0/4088873483 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596c1facf00 con 0x5596c1fa7e50 2024-04-25T01:36:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2820 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c08014fd0 tx=0x7f1c080097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bd83dbeb41af46d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.269+0000 7f1bf77fe640 1 -- 172.21.15.16:0/4088873483 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08037070 con 0x5596c1fa7e50 2024-04-25T01:36:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.270+0000 7f1bf77fe640 1 -- 172.21.15.16:0/4088873483 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c0801f8c0 con 0x5596c1fa7e50 2024-04-25T01:36:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.270+0000 7f1bf77fe640 1 -- 172.21.15.16:0/4088873483 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0802e070 con 0x5596c1fa7e50 2024-04-25T01:36:27.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:26 smithi156 ceph-mon[32631]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 55 MiB/s wr, 963 op/s 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 msgr2=0x5596c1fb2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2820 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c08014fd0 tx=0x7f1c080097b0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 shutdown_connections 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fad9b0 0x5596c1fa6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fa8840 0x5596c1fb98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 --2- 172.21.15.16:0/4088873483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa7e50 0x5596c1fb2820 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.271+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 >> 172.21.15.16:0/4088873483 conn(0x5596c1fa34c0 msgr2=0x5596c1fa6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.272+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 shutdown_connections 2024-04-25T01:36:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.272+0000 7f1c16b80740 1 -- 172.21.15.16:0/4088873483 wait complete. 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.272+0000 7f1c16b80740 1 -- start start 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.272+0000 7f1c16b80740 1 Processor -- start 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa7e50 0x5596c1fb6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa8840 0x5596c1fb6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 0x5596c2022a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf000cce0 con 0x5596c1fa7e50 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf000e150 con 0x5596c1fad9b0 2024-04-25T01:36:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf000d3b0 con 0x5596c1fa8840 2024-04-25T01:36:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c16b80740 10 client.0 _ll_register_callbacks cb 0x7f1c0800daf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c1140d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa8840 0x5596c1fb6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c1140d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa8840 0x5596c1fb6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:52316/0 (socket says 172.21.15.16:52316) 2024-04-25T01:36:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c1140d640 1 -- 172.21.15.16:0/1008627661 learned_addr learned my addr 172.21.15.16:0/1008627661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.273+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa7e50 0x5596c1fb6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1c11c0e640 1 -- 172.21.15.16:0/1008627661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa8840 msgr2=0x5596c1fb6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa8840 0x5596c1fb6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1c11c0e640 1 -- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 msgr2=0x5596c2022a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 0x5596c2022a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1c11c0e640 1 -- 172.21.15.16:0/1008627661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596c2022fa0 con 0x5596c1fa7e50 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1c11c0e640 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa7e50 0x5596c1fb6880 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1c08035040 tx=0x7f1c08032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eed9c03878793b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.274+0000 7f1bf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.275+0000 7f1bf57fa640 1 -- 172.21.15.16:0/1008627661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08037070 con 0x5596c1fa7e50 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.275+0000 7f1bf57fa640 1 -- 172.21.15.16:0/1008627661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1c08032860 con 0x5596c1fa7e50 2024-04-25T01:36:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.275+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c0801f8c0 con 0x5596c1fa7e50 2024-04-25T01:36:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.275+0000 7f1bf57fa640 1 -- 172.21.15.16:0/1008627661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1c0802e040 con 0x5596c1fa7e50 2024-04-25T01:36:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.275+0000 7f1c16b80740 10 client.15957 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.275+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5596c20242b0 con 0x5596c1fa7e50 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.276+0000 7f1bf57fa640 1 -- 172.21.15.16:0/1008627661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c0801f8c0 con 0x5596c1fa7e50 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.276+0000 7f1bf57fa640 1 -- 172.21.15.16:0/1008627661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f1c0802a540 con 0x5596c1fa7e50 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c16b80740 10 client.15957 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c16b80740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c16b80740 20 client.15957 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c16b80740 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c2024b00 0x5596c2026f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c16b80740 4 client.15957 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:36:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596c1f13540 con 0x5596c2024b00 2024-04-25T01:36:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.277+0000 7f1c1140d640 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c2024b00 0x5596c2026f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.278+0000 7f1bf57fa640 10 client.15957 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.278+0000 7f1c1140d640 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c2024b00 0x5596c2026f20 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c42e6010709affd server_cookie=9d0c11f812b1c7f8 in_seq=0 out_seq=0 2024-04-25T01:36:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.283+0000 7f1bf57fa640 1 -- 172.21.15.16:0/1008627661 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5596c1f13540 con 0x5596c2024b00 2024-04-25T01:36:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.283+0000 7f1bf57fa640 10 client.15957 handle_command_reply: tid=0 2024-04-25T01:36:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.285+0000 7f1c16b80740 1 client.15957 shutdown 2024-04-25T01:36:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.286+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa7e50 msgr2=0x5596c1fb6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.286+0000 7f1c16b80740 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa7e50 0x5596c1fb6880 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1c08035040 tx=0x7f1c08032f90 comp rx=0 tx=0).stop 2024-04-25T01:36:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 shutdown_connections 2024-04-25T01:36:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5596c2024b00 0x5596c2026f20 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596c1fad9b0 0x5596c2022a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5596c1fa8840 0x5596c1fb6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 --2- 172.21.15.16:0/1008627661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5596c1fa7e50 0x5596c1fb6880 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1bf57fa640 0 client.15957 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 >> 172.21.15.16:0/1008627661 conn(0x5596c1fa34c0 msgr2=0x5596c1fb2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 shutdown_connections 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.287+0000 7f1c16b80740 1 -- 172.21.15.16:0/1008627661 wait complete. 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.288+0000 7f1c16b80740 20 client.15957 trim_cache size 0 max 16384 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:36:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:36:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:36:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:36:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.289+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bf80a2de0 msgr2=0x7f1bf80a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.289+0000 7f1c13e72640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bf80a2de0 0x7f1bf80a52a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c08006fd0 tx=0x7f1c08008040 comp rx=0 tx=0).stop 2024-04-25T01:36:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.289+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 msgr2=0x7f1c0c118150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.289+0000 7f1c13e72640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c118150 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc02aa30 tx=0x7f1bfc030b10 comp rx=0 tx=0).stop 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 shutdown_connections 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1bf80a2de0 0x7f1bf80a52a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c0c10da60 0x7f1c0c119190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c10b0b0 0x7f1c0c118150 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 --2- 172.21.15.16:0/3709798629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0c108740 0x7f1c0c117c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 >> 172.21.15.16:0/3709798629 conn(0x7f1c0c06e7b0 msgr2=0x7f1c0c10b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.290+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 shutdown_connections 2024-04-25T01:36:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:27.291+0000 7f1c13e72640 1 -- 172.21.15.16:0/3709798629 wait complete. 2024-04-25T01:36:29.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:28 smithi038 ceph-mon[32841]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 66 MiB/s wr, 1.17k op/s 2024-04-25T01:36:29.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:28 smithi156 ceph-mon[32631]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 66 MiB/s wr, 1.17k op/s 2024-04-25T01:36:29.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:28 smithi016 ceph-mon[29153]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 66 MiB/s wr, 1.17k op/s 2024-04-25T01:36:31.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:30 smithi038 ceph-mon[32841]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 60 MiB/s wr, 1.05k op/s 2024-04-25T01:36:31.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:30 smithi156 ceph-mon[32631]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 60 MiB/s wr, 1.05k op/s 2024-04-25T01:36:31.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:30 smithi016 ceph-mon[29153]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 60 MiB/s wr, 1.05k op/s 2024-04-25T01:36:33.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:32 smithi038 ceph-mon[32841]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 60 MiB/s wr, 1.07k op/s 2024-04-25T01:36:33.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:32 smithi156 ceph-mon[32631]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 60 MiB/s wr, 1.07k op/s 2024-04-25T01:36:33.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:32 smithi016 ceph-mon[29153]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 60 MiB/s wr, 1.07k op/s 2024-04-25T01:36:35.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:34 smithi038 ceph-mon[32841]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:35.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:34 smithi156 ceph-mon[32631]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:35.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:34 smithi016 ceph-mon[29153]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 65 MiB/s wr, 1.15k op/s 2024-04-25T01:36:37.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:36 smithi016 ceph-mon[29153]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 53 MiB/s wr, 943 op/s 2024-04-25T01:36:37.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:36 smithi038 ceph-mon[32841]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 53 MiB/s wr, 943 op/s 2024-04-25T01:36:37.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:36 smithi156 ceph-mon[32631]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 53 MiB/s wr, 943 op/s 2024-04-25T01:36:39.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:38 smithi038 ceph-mon[32841]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 64 MiB/s wr, 1.14k op/s 2024-04-25T01:36:39.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:38 smithi156 ceph-mon[32631]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 64 MiB/s wr, 1.14k op/s 2024-04-25T01:36:39.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:38 smithi016 ceph-mon[29153]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 64 MiB/s wr, 1.14k op/s 2024-04-25T01:36:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 -- 172.21.15.16:0/1120531438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 msgr2=0x7f10f80962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 --2- 172.21.15.16:0/1120531438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f80962f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009860 tx=0x7f10f002a340 comp rx=0 tx=0).stop 2024-04-25T01:36:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 -- 172.21.15.16:0/1120531438 shutdown_connections 2024-04-25T01:36:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 --2- 172.21.15.16:0/1120531438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 0x7f10f809d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 --2- 172.21.15.16:0/1120531438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f80962f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 --2- 172.21.15.16:0/1120531438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 0x7f10f80950a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 -- 172.21.15.16:0/1120531438 >> 172.21.15.16:0/1120531438 conn(0x7f10f8090210 msgr2=0x7f10f8092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 -- 172.21.15.16:0/1120531438 shutdown_connections 2024-04-25T01:36:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.445+0000 7f110638c640 1 -- 172.21.15.16:0/1120531438 wait complete. 2024-04-25T01:36:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 Processor -- start 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 -- start start 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 0x7f10f8129d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f812a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 0x7f10f8131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10f809f6b0 con 0x7f10f8096830 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10f809f530 con 0x7f10f8095e70 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.446+0000 7f110638c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f10f809f830 con 0x7f10f8094c80 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f110538a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 0x7f10f8129d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f110538a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 0x7f10f8129d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:54732/0 (socket says 172.21.15.16:54732) 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f110538a640 1 -- 172.21.15.16:0/1419871930 learned_addr learned my addr 172.21.15.16:0/1419871930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f812a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 0x7f10f8131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 -- 172.21.15.16:0/1419871930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 msgr2=0x7f10f8129d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 0x7f10f8129d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 -- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 msgr2=0x7f10f8131310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 0x7f10f8131310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 -- 172.21.15.16:0/1419871930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f8131980 con 0x7f10f8095e70 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 0x7f10f8131310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f1104b89640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f812a2a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f10f002a860 tx=0x7f10f0030bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f10f67fc640 1 -- 172.21.15.16:0/1419871930 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0037050 con 0x7f10f8095e70 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f10f67fc640 1 -- 172.21.15.16:0/1419871930 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10f0038040 con 0x7f10f8095e70 2024-04-25T01:36:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.447+0000 7f10f67fc640 1 -- 172.21.15.16:0/1419871930 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00039f0 con 0x7f10f8095e70 2024-04-25T01:36:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.448+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f8131c10 con 0x7f10f8095e70 2024-04-25T01:36:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.448+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f81320b0 con 0x7f10f8095e70 2024-04-25T01:36:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.451+0000 7f10f67fc640 1 -- 172.21.15.16:0/1419871930 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f10f0045030 con 0x7f10f8095e70 2024-04-25T01:36:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.451+0000 7f110a3d8740 1 Processor -- start 2024-04-25T01:36:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.451+0000 7f110a3d8740 1 -- start start 2024-04-25T01:36:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f10f67fc640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10ec0a2b90 0x7f10ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960e3d0 0x55861960e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196138c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f10f67fc640 1 -- 172.21.15.16:0/1419871930 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f10f00eee10 con 0x7f10f8095e70 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 0x5586196163c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f110a3d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55861956dcb0 con 0x558619613f30 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f110a3d8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5586195165b0 con 0x55861960edc0 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f110a3d8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558619569c70 con 0x55861960e3d0 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f1105b8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 0x5586196163c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f1105b8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 0x5586196163c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50598/0 (socket says 172.21.15.16:50598) 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f1105b8b640 1 -- 172.21.15.16:0/1417335707 learned_addr learned my addr 172.21.15.16:0/1417335707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f1104b89640 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.452+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10ec0a2b90 0x7f10ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f110538a640 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960e3d0 0x55861960e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 -- 172.21.15.16:0/1417335707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960e3d0 msgr2=0x55861960e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960e3d0 0x55861960e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 -- 172.21.15.16:0/1417335707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 msgr2=0x5586196138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 -- 172.21.15.16:0/1417335707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558619282210 con 0x558619613f30 2024-04-25T01:36:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 0x5586196163c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1100050e00 tx=0x7f1100087c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff543495e57d805 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.453+0000 7f1105b8b640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10ec0a2b90 0x7f10ec0a5050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f11000502b0 tx=0x7f110006a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.454+0000 7f10eb7fe640 1 -- 172.21.15.16:0/1417335707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100090070 con 0x558619613f30 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.454+0000 7f10eb7fe640 1 -- 172.21.15.16:0/1417335707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f110006a070 con 0x558619613f30 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.454+0000 7f10eb7fe640 1 -- 172.21.15.16:0/1417335707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100089d70 con 0x558619613f30 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 -- 172.21.15.16:0/1417335707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 msgr2=0x5586196163c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 0x5586196163c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1100050e00 tx=0x7f1100087c60 comp rx=0 tx=0).stop 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 -- 172.21.15.16:0/1417335707 shutdown_connections 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558619613f30 0x5586196163c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1417335707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960e3d0 0x55861960e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.455+0000 7f110a3d8740 1 -- 172.21.15.16:0/1417335707 >> 172.21.15.16:0/1417335707 conn(0x558619609a40 msgr2=0x558619609e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.456+0000 7f110a3d8740 1 -- 172.21.15.16:0/1417335707 shutdown_connections 2024-04-25T01:36:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.456+0000 7f110a3d8740 1 -- 172.21.15.16:0/1417335707 wait complete. 2024-04-25T01:36:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.456+0000 7f110a3d8740 1 -- start start 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.456+0000 7f110a3d8740 1 Processor -- start 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.457+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619606400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.457+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619606940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.457+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619607950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f110a3d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558619616e80 con 0x55861960e3d0 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f1105b8b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619607950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f1105b8b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619607950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:40340/0 (socket says 172.21.15.16:40340) 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f1105b8b640 1 -- 172.21.15.16:0/3942843524 learned_addr learned my addr 172.21.15.16:0/3942843524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558619529b50 con 0x558619613f30 2024-04-25T01:36:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558619617000 con 0x55861960edc0 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f110a3d8740 10 client.0 _ll_register_callbacks cb 0x558619564ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f1105b8b640 1 -- 172.21.15.16:0/3942843524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 msgr2=0x558619606940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f1105b8b640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619606940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f1105b8b640 1 -- 172.21.15.16:0/3942843524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 msgr2=0x558619606400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.458+0000 7f110538a640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619606400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f1105b8b640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619606400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f1105b8b640 1 -- 172.21.15.16:0/3942843524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55861934e620 con 0x558619613f30 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f110006a070 con 0x558619613f30 2024-04-25T01:36:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f110538a640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619606400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f1105b8b640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619607950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f11000648b0 tx=0x7f1100067a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f525c13f2ed9e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f10e8ff9640 10 client.25634 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f10e8ff9640 1 -- 172.21.15.16:0/3942843524 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000a4050 con 0x558619613f30 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.459+0000 7f10e8ff9640 1 -- 172.21.15.16:0/3942843524 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1100089990 con 0x558619613f30 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.460+0000 7f10e8ff9640 1 -- 172.21.15.16:0/3942843524 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1100090040 con 0x558619613f30 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.460+0000 7f110a3d8740 10 client.25634 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.460+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=22}) -- 0x5586196274a0 con 0x558619613f30 2024-04-25T01:36:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f10e8ff9640 1 -- 172.21.15.16:0/3942843524 <== mon.2 v2:172.21.15.38:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f110008a460 con 0x558619613f30 2024-04-25T01:36:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110a3d8740 10 client.25634 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110a3d8740 10 client.25634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110a3d8740 20 client.25634 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110a3d8740 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558619628e80 0x55861962b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110a3d8740 4 client.25634 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:40.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55861957b1f0 con 0x558619628e80 2024-04-25T01:36:40.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.461+0000 7f110538a640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558619628e80 0x55861962b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.462+0000 7f110538a640 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558619628e80 0x55861962b2a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd5254d75b399c2e server_cookie=cd1ef767a390dd4f in_seq=0 out_seq=0 2024-04-25T01:36:40.439 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.462+0000 7f10e8ff9640 10 client.25634 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.464+0000 7f10e8ff9640 1 -- 172.21.15.16:0/3942843524 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55861957b1f0 con 0x558619628e80 2024-04-25T01:36:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.464+0000 7f10e8ff9640 10 client.25634 handle_command_reply: tid=0 2024-04-25T01:36:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.464+0000 7f110a3d8740 1 client.25634 shutdown 2024-04-25T01:36:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.465+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 msgr2=0x558619607950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.465+0000 7f110a3d8740 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619607950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f11000648b0 tx=0x7f1100067a10 comp rx=0 tx=0).stop 2024-04-25T01:36:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.469+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 shutdown_connections 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.469+0000 7f110a3d8740 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x558619628e80 0x55861962b2a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.469+0000 7f110a3d8740 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619607950 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.469+0000 7f110a3d8740 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619606940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.469+0000 7f110a3d8740 1 --2- 172.21.15.16:0/3942843524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619606400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.469+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 >> 172.21.15.16:0/3942843524 conn(0x558619609a40 msgr2=0x55861960ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.470+0000 7f10e8ff9640 0 client.25634 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.470+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 shutdown_connections 2024-04-25T01:36:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.471+0000 7f110a3d8740 1 -- 172.21.15.16:0/3942843524 wait complete. 2024-04-25T01:36:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.473+0000 7f110a3d8740 20 client.25634 trim_cache size 0 max 16384 2024-04-25T01:36:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.505+0000 7f110a3d8740 1 Processor -- start 2024-04-25T01:36:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.505+0000 7f110a3d8740 1 -- start start 2024-04-25T01:36:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.506+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619610af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.506+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619611030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.506+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619606400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110a3d8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10f813d820 con 0x55861960e3d0 2024-04-25T01:36:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110a3d8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10f813b0f0 con 0x558619613f30 2024-04-25T01:36:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110a3d8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8138f40 con 0x55861960edc0 2024-04-25T01:36:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110538a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619610af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110538a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619610af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50612/0 (socket says 172.21.15.16:50612) 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110538a640 1 -- 172.21.15.16:0/1585155852 learned_addr learned my addr 172.21.15.16:0/1585155852 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619611030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 -- 172.21.15.16:0/1585155852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 msgr2=0x558619606400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619606400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 -- 172.21.15.16:0/1585155852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 msgr2=0x558619610af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619610af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 -- 172.21.15.16:0/1585155852 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558619610580 con 0x55861960edc0 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f110538a640 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619610af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.507+0000 7f1104b89640 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619611030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f10f00bb610 tx=0x7f10f00b79e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c2ae92a19b98aed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.508+0000 7f10eaffd640 1 -- 172.21.15.16:0/1585155852 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00720e0 con 0x55861960edc0 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.508+0000 7f10eaffd640 1 -- 172.21.15.16:0/1585155852 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10f00bbd80 con 0x55861960edc0 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.508+0000 7f10eaffd640 1 -- 172.21.15.16:0/1585155852 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00eca90 con 0x55861960edc0 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 -- 172.21.15.16:0/1585155852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 msgr2=0x558619611030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619611030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f10f00bb610 tx=0x7f10f00b79e0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 -- 172.21.15.16:0/1585155852 shutdown_connections 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558619613f30 0x558619606400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55861960edc0 0x558619611030 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 --2- 172.21.15.16:0/1585155852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619610af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 -- 172.21.15.16:0/1585155852 >> 172.21.15.16:0/1585155852 conn(0x558619609a40 msgr2=0x558619603fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 -- 172.21.15.16:0/1585155852 shutdown_connections 2024-04-25T01:36:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.509+0000 7f110a3d8740 1 -- 172.21.15.16:0/1585155852 wait complete. 2024-04-25T01:36:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.510+0000 7f110a3d8740 1 -- start start 2024-04-25T01:36:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.510+0000 7f110a3d8740 1 Processor -- start 2024-04-25T01:36:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619622280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196227c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110538a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619622280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558619613f30 0x55861968d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110538a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619622280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50618/0 (socket says 172.21.15.16:50618) 2024-04-25T01:36:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81395b0 con 0x55861960e3d0 2024-04-25T01:36:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110538a640 1 -- 172.21.15.16:0/155131862 learned_addr learned my addr 172.21.15.16:0/155131862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10f813aa20 con 0x55861960edc0 2024-04-25T01:36:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8139c80 con 0x558619613f30 2024-04-25T01:36:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f110a3d8740 10 client.0 _ll_register_callbacks cb 0x7f110008e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f1104b89640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.511+0000 7f1105b8b640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558619613f30 0x55861968d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110538a640 1 -- 172.21.15.16:0/155131862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558619613f30 msgr2=0x55861968d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110538a640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558619613f30 0x55861968d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110538a640 1 -- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 msgr2=0x5586196227c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110538a640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196227c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110538a640 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586196257d0 con 0x55861960e3d0 2024-04-25T01:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110538a640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619622280 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009430 tx=0x7f10fc009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2e5c7948bc6b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f10e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f10e8ff9640 1 -- 172.21.15.16:0/155131862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc01d420 con 0x55861960e3d0 2024-04-25T01:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f10e8ff9640 1 -- 172.21.15.16:0/155131862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f10fc01d5c0 con 0x55861960e3d0 2024-04-25T01:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f00bbd80 con 0x55861960e3d0 2024-04-25T01:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.512+0000 7f10e8ff9640 1 -- 172.21.15.16:0/155131862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f10fc02c050 con 0x55861960e3d0 2024-04-25T01:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 10 client.15972 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55861968e9e0 con 0x55861960e3d0 2024-04-25T01:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f10e8ff9640 1 -- 172.21.15.16:0/155131862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10f00bbd80 con 0x55861960e3d0 2024-04-25T01:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f10e8ff9640 1 -- 172.21.15.16:0/155131862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f10fc0253c0 con 0x55861960e3d0 2024-04-25T01:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 10 client.15972 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 20 client.15972 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55861968f3e0 0x558619691800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.513+0000 7f110a3d8740 4 client.15972 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.514+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558619625ab0 con 0x55861968f3e0 2024-04-25T01:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.514+0000 7f1104b89640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55861968f3e0 0x558619691800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.514+0000 7f10e8ff9640 10 client.15972 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.514+0000 7f1104b89640 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55861968f3e0 0x558619691800 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a750a496218b0e2c server_cookie=5709b1cb9ec9e733 in_seq=0 out_seq=0 2024-04-25T01:36:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.540+0000 7f10e8ff9640 1 -- 172.21.15.16:0/155131862 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558619625ab0 con 0x55861968f3e0 2024-04-25T01:36:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.540+0000 7f10e8ff9640 10 client.15972 handle_command_reply: tid=0 2024-04-25T01:36:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.540+0000 7f110a3d8740 1 client.15972 shutdown 2024-04-25T01:36:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 msgr2=0x558619622280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619622280 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009430 tx=0x7f10fc009ed0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 shutdown_connections 2024-04-25T01:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558619613f30 0x55861968d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55861968f3e0 0x558619691800 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55861960edc0 0x5586196227c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 --2- 172.21.15.16:0/155131862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55861960e3d0 0x558619622280 secure :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009430 tx=0x7f10fc009ed0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 >> 172.21.15.16:0/155131862 conn(0x558619609a40 msgr2=0x5586196063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.542+0000 7f10e8ff9640 0 client.15972 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.545+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 shutdown_connections 2024-04-25T01:36:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.546+0000 7f110a3d8740 1 -- 172.21.15.16:0/155131862 wait complete. 2024-04-25T01:36:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.546+0000 7f110a3d8740 20 client.15972 trim_cache size 0 max 16384 2024-04-25T01:36:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:36:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:36:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:36:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:36:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10ec0a2b90 msgr2=0x7f10ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10ec0a2b90 0x7f10ec0a5050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f11000502b0 tx=0x7f110006a000 comp rx=0 tx=0).stop 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 msgr2=0x7f10f812a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f812a2a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f10f002a860 tx=0x7f10f0030bd0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 shutdown_connections 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10ec0a2b90 0x7f10ec0a5050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f8096830 0x7f10f8131310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.549+0000 7f110638c640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8095e70 0x7f10f812a2a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.550+0000 7f110638c640 1 --2- 172.21.15.16:0/1419871930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10f8094c80 0x7f10f8129d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.550+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 >> 172.21.15.16:0/1419871930 conn(0x7f10f8090210 msgr2=0x7f10f80991f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.550+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 shutdown_connections 2024-04-25T01:36:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:40.550+0000 7f110638c640 1 -- 172.21.15.16:0/1419871930 wait complete. 2024-04-25T01:36:41.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:40 smithi038 ceph-mon[32841]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:36:41.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:40 smithi156 ceph-mon[32631]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:36:41.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:40 smithi016 ceph-mon[29153]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:36:43.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:42 smithi038 ceph-mon[32841]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:36:43.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:42 smithi156 ceph-mon[32631]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:36:43.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:42 smithi016 ceph-mon[29153]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:36:45.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:44 smithi038 ceph-mon[32841]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 63 MiB/s wr, 1.14k op/s 2024-04-25T01:36:45.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:44 smithi156 ceph-mon[32631]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 63 MiB/s wr, 1.14k op/s 2024-04-25T01:36:45.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:44 smithi016 ceph-mon[29153]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 63 MiB/s wr, 1.14k op/s 2024-04-25T01:36:47.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:46 smithi038 ceph-mon[32841]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 52 MiB/s wr, 946 op/s 2024-04-25T01:36:47.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:46 smithi156 ceph-mon[32631]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 52 MiB/s wr, 946 op/s 2024-04-25T01:36:47.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:46 smithi016 ceph-mon[29153]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 52 MiB/s wr, 946 op/s 2024-04-25T01:36:49.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:48 smithi038 ceph-mon[32841]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 64 MiB/s wr, 1.16k op/s 2024-04-25T01:36:49.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:48 smithi156 ceph-mon[32631]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 64 MiB/s wr, 1.16k op/s 2024-04-25T01:36:49.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:48 smithi016 ceph-mon[29153]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 64 MiB/s wr, 1.16k op/s 2024-04-25T01:36:51.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:50 smithi038 ceph-mon[32841]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 58 MiB/s wr, 1.05k op/s 2024-04-25T01:36:51.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:50 smithi156 ceph-mon[32631]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 58 MiB/s wr, 1.05k op/s 2024-04-25T01:36:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:50 smithi016 ceph-mon[29153]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 58 MiB/s wr, 1.05k op/s 2024-04-25T01:36:53.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:52 smithi038 ceph-mon[32841]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 59 MiB/s wr, 1.06k op/s 2024-04-25T01:36:53.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:52 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:53.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:52 smithi156 ceph-mon[32631]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 59 MiB/s wr, 1.06k op/s 2024-04-25T01:36:53.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:52 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:53.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:52 smithi016 ceph-mon[29153]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 59 MiB/s wr, 1.06k op/s 2024-04-25T01:36:53.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:52 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.722+0000 7fafaeced640 1 -- 172.21.15.16:0/1330783319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 msgr2=0x7fafa00a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.722+0000 7fafaeced640 1 --2- 172.21.15.16:0/1330783319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa00a5400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009900 tx=0x7fafa402a410 comp rx=0 tx=0).stop 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 -- 172.21.15.16:0/1330783319 shutdown_connections 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 --2- 172.21.15.16:0/1330783319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a6b90 0x7fafa00ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 --2- 172.21.15.16:0/1330783319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa00a61d0 0x7fafa00a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 --2- 172.21.15.16:0/1330783319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa00a5400 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 -- 172.21.15.16:0/1330783319 >> 172.21.15.16:0/1330783319 conn(0x7fafa00a0530 msgr2=0x7fafa00a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 -- 172.21.15.16:0/1330783319 shutdown_connections 2024-04-25T01:36:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.723+0000 7fafaeced640 1 -- 172.21.15.16:0/1330783319 wait complete. 2024-04-25T01:36:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 Processor -- start 2024-04-25T01:36:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 -- start start 2024-04-25T01:36:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa013f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a61d0 0x7fafa013f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa00a6b90 0x7fafa01467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafa00af2e0 con 0x7fafa00a6b90 2024-04-25T01:36:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fafa00af160 con 0x7fafa00a61d0 2024-04-25T01:36:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafaeced640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fafa00af460 con 0x7fafa00a4fe0 2024-04-25T01:36:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafadceb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa013f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafadceb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa013f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44070/0 (socket says 172.21.15.16:44070) 2024-04-25T01:36:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.724+0000 7fafadceb640 1 -- 172.21.15.16:0/4179639219 learned_addr learned my addr 172.21.15.16:0/4179639219 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafad4ea640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a61d0 0x7fafa013f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafadceb640 1 -- 172.21.15.16:0/4179639219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a61d0 msgr2=0x7fafa013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafadceb640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a61d0 0x7fafa013f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafadceb640 1 -- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa00a6b90 msgr2=0x7fafa01467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafadceb640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa00a6b90 0x7fafa01467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafadceb640 1 -- 172.21.15.16:0/4179639219 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa0146e40 con 0x7fafa00a4fe0 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafad4ea640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a61d0 0x7fafa013f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafadceb640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa013f270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fafa40314e0 tx=0x7fafa4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7faf9effd640 1 -- 172.21.15.16:0/4179639219 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa403f070 con 0x7fafa00a4fe0 2024-04-25T01:36:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.725+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa00b1cc0 con 0x7fafa00a4fe0 2024-04-25T01:36:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.726+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa00b2200 con 0x7fafa00a4fe0 2024-04-25T01:36:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.726+0000 7faf9effd640 1 -- 172.21.15.16:0/4179639219 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa4031c30 con 0x7fafa00a4fe0 2024-04-25T01:36:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.726+0000 7faf9effd640 1 -- 172.21.15.16:0/4179639219 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4035030 con 0x7fafa00a4fe0 2024-04-25T01:36:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 Processor -- start 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 -- start start 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a934b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a93b20 0x55e0c6a95fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0c69f1cb0 con 0x55e0c6a93b20 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e0c699a5b0 con 0x55e0c6a8dfc0 2024-04-25T01:36:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.731+0000 7fafb2c56740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e0c69edc70 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.732+0000 7fafae4ec640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.732+0000 7fafadceb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a934b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a934b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44082/0 (socket says 172.21.15.16:44082) 2024-04-25T01:36:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 -- 172.21.15.16:0/2608691461 learned_addr learned my addr 172.21.15.16:0/2608691461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 -- 172.21.15.16:0/2608691461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8dfc0 msgr2=0x55e0c6a8e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7faf9effd640 1 -- 172.21.15.16:0/4179639219 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fafa4042070 con 0x7fafa00a4fe0 2024-04-25T01:36:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 -- 172.21.15.16:0/2608691461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a93b20 msgr2=0x55e0c6a95fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:36:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a93b20 0x55e0c6a95fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.733+0000 7fafadceb640 1 -- 172.21.15.16:0/2608691461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c6706210 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.734+0000 7faf9effd640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf940a2c90 0x7faf940a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.734+0000 7faf9effd640 1 -- 172.21.15.16:0/4179639219 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fafa40f07f0 con 0x7fafa00a4fe0 2024-04-25T01:36:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.735+0000 7fafae4ec640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf940a2c90 0x7faf940a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.735+0000 7fafadceb640 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a934b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafa40ee1b0 tx=0x7fafa402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403a9e82900fe1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.738+0000 7faf93fff640 1 -- 172.21.15.16:0/2608691461 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4049070 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.738+0000 7faf93fff640 1 -- 172.21.15.16:0/2608691461 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa40bd6d0 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.739+0000 7faf93fff640 1 -- 172.21.15.16:0/2608691461 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa40bcfd0 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafb2c56740 1 -- 172.21.15.16:0/2608691461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 msgr2=0x55e0c6a934b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafb2c56740 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a934b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafa40ee1b0 tx=0x7fafa402a410 comp rx=0 tx=0).stop 2024-04-25T01:36:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafb2c56740 1 -- 172.21.15.16:0/2608691461 shutdown_connections 2024-04-25T01:36:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafb2c56740 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a93b20 0x55e0c6a95fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafb2c56740 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a934b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafae4ec640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf940a2c90 0x7faf940a5150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fafa8067020 tx=0x7fafa8073040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.740+0000 7fafb2c56740 1 --2- 172.21.15.16:0/2608691461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.741+0000 7fafb2c56740 1 -- 172.21.15.16:0/2608691461 >> 172.21.15.16:0/2608691461 conn(0x55e0c6a89630 msgr2=0x55e0c6a89a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:53.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.741+0000 7fafb2c56740 1 -- 172.21.15.16:0/2608691461 shutdown_connections 2024-04-25T01:36:53.717 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.742+0000 7fafb2c56740 1 -- 172.21.15.16:0/2608691461 wait complete. 2024-04-25T01:36:53.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.742+0000 7fafb2c56740 1 -- start start 2024-04-25T01:36:53.718 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.743+0000 7fafb2c56740 1 Processor -- start 2024-04-25T01:36:53.719 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.743+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a98060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.743+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a985a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafb2c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6a96890 con 0x55e0c6a8dfc0 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafad4ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafad4ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:53364/0 (socket says 172.21.15.16:53364) 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafad4ea640 1 -- 172.21.15.16:0/778903107 learned_addr learned my addr 172.21.15.16:0/778903107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafadceb640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a98060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.720 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e0c69adb50 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafae4ec640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6a96a10 con 0x55e0c6a93b20 2024-04-25T01:36:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.744+0000 7fafb2c56740 10 client.0 _ll_register_callbacks cb 0x55e0c69e8ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7fafae4ec640 1 -- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 msgr2=0x55e0c6a985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7fafae4ec640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.721 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7fafae4ec640 1 -- 172.21.15.16:0/778903107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8dfc0 msgr2=0x55e0c6a98060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7fafae4ec640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a98060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7fafae4ec640 1 -- 172.21.15.16:0/778903107 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c67d2620 con 0x55e0c6a93b20 2024-04-25T01:36:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7fafae4ec640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9f5e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafa804f7d0 tx=0x7fafa8067890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9fe683cf2e8c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7faf917fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:36:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7faf917fa640 1 -- 172.21.15.16:0/778903107 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa80875f0 con 0x55e0c6a93b20 2024-04-25T01:36:53.722 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.745+0000 7faf917fa640 1 -- 172.21.15.16:0/778903107 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fafa80863a0 con 0x55e0c6a93b20 2024-04-25T01:36:53.725 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.750+0000 7fafadceb640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a98060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:53.726 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.750+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafa40bd6d0 con 0x55e0c6a93b20 2024-04-25T01:36:53.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.751+0000 7faf917fa640 1 -- 172.21.15.16:0/778903107 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fafa8087790 con 0x55e0c6a93b20 2024-04-25T01:36:53.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.753+0000 7fafb2c56740 10 client.25609 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:53.729 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.753+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55e0c6aa2950 con 0x55e0c6a93b20 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7faf917fa640 1 -- 172.21.15.16:0/778903107 <== mon.1 v2:172.21.15.156:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fafa80865e0 con 0x55e0c6a93b20 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafb2c56740 10 client.25609 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafb2c56740 10 client.25609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafb2c56740 20 client.25609 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafb2c56740 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6aa3f80 0x55e0c6aa6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafb2c56740 4 client.25609 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0c69ff1f0 con 0x55e0c6aa3f80 2024-04-25T01:36:53.730 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.754+0000 7fafadceb640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6aa3f80 0x55e0c6aa6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.755+0000 7fafadceb640 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6aa3f80 0x55e0c6aa6420 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457105bc689c2a2b server_cookie=f65873e978e5ccd2 in_seq=0 out_seq=0 2024-04-25T01:36:53.731 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.755+0000 7faf917fa640 10 client.25609 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.756+0000 7faf917fa640 1 -- 172.21.15.16:0/778903107 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0c69ff1f0 con 0x55e0c6aa3f80 2024-04-25T01:36:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.756+0000 7faf917fa640 10 client.25609 handle_command_reply: tid=0 2024-04-25T01:36:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.757+0000 7fafb2c56740 1 client.25609 shutdown 2024-04-25T01:36:53.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.757+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 msgr2=0x55e0c6a9f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.757+0000 7fafb2c56740 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9f5e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafa804f7d0 tx=0x7fafa8067890 comp rx=0 tx=0).stop 2024-04-25T01:36:53.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.757+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 shutdown_connections 2024-04-25T01:36:53.733 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.757+0000 7fafb2c56740 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9f5e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.757+0000 7fafb2c56740 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6aa3f80 0x55e0c6aa6420 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.758+0000 7fafb2c56740 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.758+0000 7faf917fa640 0 client.25609 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:53.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.758+0000 7fafb2c56740 1 --2- 172.21.15.16:0/778903107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a98060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.734 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.758+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 >> 172.21.15.16:0/778903107 conn(0x55e0c6a89630 msgr2=0x55e0c6a88830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.767+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 shutdown_connections 2024-04-25T01:36:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.768+0000 7fafb2c56740 1 -- 172.21.15.16:0/778903107 wait complete. 2024-04-25T01:36:53.746 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.770+0000 7fafb2c56740 20 client.25609 trim_cache size 0 max 16384 2024-04-25T01:36:53.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.812+0000 7fafb2c56740 1 Processor -- start 2024-04-25T01:36:53.788 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.813+0000 7fafb2c56740 1 -- start start 2024-04-25T01:36:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.813+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.813+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.813+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a93b20 0x55e0c6a8c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.814+0000 7fafb2c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c010d50 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.814+0000 7fafb2c56740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c00e620 con 0x55e0c6a93b20 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.814+0000 7fafae4ec640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a93b20 0x55e0c6a8c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.814+0000 7fafae4ec640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a93b20 0x55e0c6a8c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:53374/0 (socket says 172.21.15.16:53374) 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.814+0000 7fafae4ec640 1 -- 172.21.15.16:0/4285041646 learned_addr learned my addr 172.21.15.16:0/4285041646 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.814+0000 7fafb2c56740 1 -- 172.21.15.16:0/4285041646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c00c470 con 0x55e0c6a8dfc0 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafadceb640 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafad4ea640 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafad4ea640 1 -- 172.21.15.16:0/4285041646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8dfc0 msgr2=0x55e0c6a8c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafad4ea640 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafad4ea640 1 -- 172.21.15.16:0/4285041646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a93b20 msgr2=0x55e0c6a8c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafad4ea640 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a93b20 0x55e0c6a8c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.815+0000 7fafad4ea640 1 -- 172.21.15.16:0/4285041646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c6a988f0 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.816+0000 7fafad4ea640 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9f5a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7faf98003800 tx=0x7faf98003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1860752d67d8824f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.816+0000 7faf937fe640 1 -- 172.21.15.16:0/4285041646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98011bc0 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.816+0000 7faf937fe640 1 -- 172.21.15.16:0/4285041646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf98003e50 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.816+0000 7faf937fe640 1 -- 172.21.15.16:0/4285041646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98004d80 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.817+0000 7fafb2c56740 1 -- 172.21.15.16:0/4285041646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8e9b0 msgr2=0x55e0c6a9f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.817+0000 7fafb2c56740 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9f5a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7faf98003800 tx=0x7faf98003c00 comp rx=0 tx=0).stop 2024-04-25T01:36:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.818+0000 7fafb2c56740 1 -- 172.21.15.16:0/4285041646 shutdown_connections 2024-04-25T01:36:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.818+0000 7fafb2c56740 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a93b20 0x55e0c6a8c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.818+0000 7fafb2c56740 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9f5a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.818+0000 7fafb2c56740 1 --2- 172.21.15.16:0/4285041646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a8dfc0 0x55e0c6a8c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.818+0000 7fafb2c56740 1 -- 172.21.15.16:0/4285041646 >> 172.21.15.16:0/4285041646 conn(0x55e0c6a89630 msgr2=0x55e0c6a8fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.819+0000 7fafb2c56740 1 -- 172.21.15.16:0/4285041646 shutdown_connections 2024-04-25T01:36:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.819+0000 7fafb2c56740 1 -- 172.21.15.16:0/4285041646 wait complete. 2024-04-25T01:36:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.819+0000 7fafb2c56740 1 -- start start 2024-04-25T01:36:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.819+0000 7fafb2c56740 1 Processor -- start 2024-04-25T01:36:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 0x55e0c6a9e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c00cae0 con 0x55e0c6aa3f80 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c00df50 con 0x55e0c6a8e9b0 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c00d1b0 con 0x55e0c6a93b20 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafb2c56740 10 client.0 _ll_register_callbacks cb 0x7fafa807ca60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafae4ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 0x55e0c6a9e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafae4ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 0x55e0c6a9e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59204/0 (socket says 172.21.15.16:59204) 2024-04-25T01:36:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.820+0000 7fafae4ec640 1 -- 172.21.15.16:0/3269638835 learned_addr learned my addr 172.21.15.16:0/3269638835 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:36:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.821+0000 7fafae4ec640 1 -- 172.21.15.16:0/3269638835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 msgr2=0x55e0c6a9d190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:36:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.821+0000 7fafae4ec640 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.821+0000 7fafae4ec640 1 -- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 msgr2=0x55e0c6a9cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:36:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.821+0000 7fafae4ec640 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.821+0000 7fafae4ec640 1 -- 172.21.15.16:0/3269638835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c6a9e790 con 0x55e0c6aa3f80 2024-04-25T01:36:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.821+0000 7fafae4ec640 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 0x55e0c6a9e1a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fafa8067720 tx=0x7fafa807b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4aa14e52213049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.822+0000 7faf917fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:36:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.822+0000 7faf917fa640 1 -- 172.21.15.16:0/3269638835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8061d80 con 0x55e0c6aa3f80 2024-04-25T01:36:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.822+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf98003e50 con 0x55e0c6aa3f80 2024-04-25T01:36:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.822+0000 7faf917fa640 1 -- 172.21.15.16:0/3269638835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fafa80863a0 con 0x55e0c6aa3f80 2024-04-25T01:36:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.823+0000 7faf917fa640 1 -- 172.21.15.16:0/3269638835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fafa80865a0 con 0x55e0c6aa3f80 2024-04-25T01:36:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.825+0000 7fafb2c56740 10 client.15981 fetch_fsmap learned FSMap version 22 2024-04-25T01:36:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.825+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e0c6b0a5a0 con 0x55e0c6aa3f80 2024-04-25T01:36:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7faf917fa640 1 -- 172.21.15.16:0/3269638835 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf98003e50 con 0x55e0c6aa3f80 2024-04-25T01:36:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7faf917fa640 1 -- 172.21.15.16:0/3269638835 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fafa809a680 con 0x55e0c6aa3f80 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7fafb2c56740 10 client.15981 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7fafb2c56740 10 client.15981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7fafb2c56740 20 client.15981 populate_metadata read hostname 'smithi016' 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7fafb2c56740 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6b0b000 0x55e0c6b0d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7fafb2c56740 4 client.15981 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.829+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0c69f9540 con 0x55e0c6b0b000 2024-04-25T01:36:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.830+0000 7fafadceb640 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6b0b000 0x55e0c6b0d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.832+0000 7fafadceb640 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6b0b000 0x55e0c6b0d440 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbf3efb1e82ed18f server_cookie=f8dbafa25a64bde0 in_seq=0 out_seq=0 2024-04-25T01:36:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.832+0000 7faf917fa640 10 client.15981 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.857+0000 7faf917fa640 1 -- 172.21.15.16:0/3269638835 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0c69f9540 con 0x55e0c6b0b000 2024-04-25T01:36:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.857+0000 7faf917fa640 10 client.15981 handle_command_reply: tid=0 2024-04-25T01:36:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.857+0000 7fafb2c56740 1 client.15981 shutdown 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 msgr2=0x55e0c6a9e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 0x55e0c6a9e1a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fafa8067720 tx=0x7fafa807b000 comp rx=0 tx=0).stop 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 shutdown_connections 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e0c6aa3f80 0x55e0c6a9e1a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0c6a93b20 0x55e0c6a9d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55e0c6b0b000 0x55e0c6b0d440 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 --2- 172.21.15.16:0/3269638835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0c6a8e9b0 0x55e0c6a9cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 >> 172.21.15.16:0/3269638835 conn(0x55e0c6a89630 msgr2=0x55e0c6a8fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7faf917fa640 0 client.15981 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 shutdown_connections 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 1 -- 172.21.15.16:0/3269638835 wait complete. 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.859+0000 7fafb2c56740 20 client.15981 trim_cache size 0 max 16384 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:36:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:36:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.860+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf940a2c90 msgr2=0x7faf940a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.860+0000 7fafaeced640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf940a2c90 0x7faf940a5150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fafa8067020 tx=0x7fafa8073040 comp rx=0 tx=0).stop 2024-04-25T01:36:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.860+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 msgr2=0x7fafa013f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.860+0000 7fafaeced640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa013f270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fafa40314e0 tx=0x7fafa4003c10 comp rx=0 tx=0).stop 2024-04-25T01:36:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.860+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 shutdown_connections 2024-04-25T01:36:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.860+0000 7fafaeced640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faf940a2c90 0x7faf940a5150 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.861+0000 7fafaeced640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa00a6b90 0x7fafa01467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.861+0000 7fafaeced640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafa00a61d0 0x7fafa013f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.861+0000 7fafaeced640 1 --2- 172.21.15.16:0/4179639219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafa00a4fe0 0x7fafa013f270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.861+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 >> 172.21.15.16:0/4179639219 conn(0x7fafa00a0530 msgr2=0x7fafa00a11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.861+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 shutdown_connections 2024-04-25T01:36:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:36:53.861+0000 7fafaeced640 1 -- 172.21.15.16:0/4179639219 wait complete. 2024-04-25T01:36:55.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:54 smithi038 ceph-mon[32841]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 64 MiB/s wr, 1.15k op/s 2024-04-25T01:36:55.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:55.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:55.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:36:55.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:54 smithi156 ceph-mon[32631]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 64 MiB/s wr, 1.15k op/s 2024-04-25T01:36:55.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:55.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:55.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:36:55.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:54 smithi016 ceph-mon[29153]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 64 MiB/s wr, 1.15k op/s 2024-04-25T01:36:55.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:55.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:55.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:36:57.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:56 smithi038 ceph-mon[32841]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 53 MiB/s wr, 952 op/s 2024-04-25T01:36:57.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:56 smithi156 ceph-mon[32631]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 53 MiB/s wr, 952 op/s 2024-04-25T01:36:57.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:56 smithi016 ceph-mon[29153]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 53 MiB/s wr, 952 op/s 2024-04-25T01:36:59.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:36:58 smithi038 ceph-mon[32841]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 64 MiB/s wr, 1.15k op/s 2024-04-25T01:36:59.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:36:58 smithi156 ceph-mon[32631]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 64 MiB/s wr, 1.15k op/s 2024-04-25T01:36:59.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:36:58 smithi016 ceph-mon[29153]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 64 MiB/s wr, 1.15k op/s 2024-04-25T01:37:01.212 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:00 smithi038 ceph-mon[32841]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-04-25T01:37:01.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:00 smithi156 ceph-mon[32631]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-04-25T01:37:01.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:00 smithi016 ceph-mon[29153]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-04-25T01:37:03.213 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:02 smithi038 ceph-mon[32841]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:37:03.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:02 smithi156 ceph-mon[32631]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:37:03.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:02 smithi016 ceph-mon[29153]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:37:05.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:05 smithi016 ceph-mon[29153]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 63 MiB/s wr, 1.13k op/s 2024-04-25T01:37:05.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:05 smithi038 ceph-mon[32841]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 63 MiB/s wr, 1.13k op/s 2024-04-25T01:37:05.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:05 smithi156 ceph-mon[32631]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 63 MiB/s wr, 1.13k op/s 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.029+0000 7f27ba1dc640 1 -- 172.21.15.16:0/2964696603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 msgr2=0x7f27b4110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.029+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/2964696603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 0x7f27b4110ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009900 tx=0x7f27a402a440 comp rx=0 tx=0).stop 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.030+0000 7f27ba1dc640 1 -- 172.21.15.16:0/2964696603 shutdown_connections 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.030+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/2964696603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 0x7f27b4110ea0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.030+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/2964696603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27b410c180 0x7f27b410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.030+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/2964696603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b4109850 0x7f27b410bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.030+0000 7f27ba1dc640 1 -- 172.21.15.16:0/2964696603 >> 172.21.15.16:0/2964696603 conn(0x7f27b406e7f0 msgr2=0x7f27b4070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.031+0000 7f27ba1dc640 1 -- 172.21.15.16:0/2964696603 shutdown_connections 2024-04-25T01:37:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.031+0000 7f27ba1dc640 1 -- 172.21.15.16:0/2964696603 wait complete. 2024-04-25T01:37:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.031+0000 7f27ba1dc640 1 Processor -- start 2024-04-25T01:37:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 -- start start 2024-04-25T01:37:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27b4109850 0x7f27b419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 0x7f27b419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 0x7f27b419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4114890 con 0x7f27b410eab0 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4114710 con 0x7f27b4109850 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27ba1dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4114a10 con 0x7f27b410c180 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 0x7f27b419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 0x7f27b419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:41692/0 (socket says 172.21.15.16:41692) 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.032+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3454607493 learned_addr learned my addr 172.21.15.16:0/3454607493 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.033+0000 7f27b3fff640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 0x7f27b419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.033+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3454607493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27b4109850 msgr2=0x7f27b419d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.033+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27b4109850 0x7f27b419d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.033+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 msgr2=0x7f27b419a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.033+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 0x7f27b419a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.033+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3454607493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b419ad10 con 0x7f27b410c180 2024-04-25T01:37:07.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.034+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 0x7f27b419de20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00b9e0 tx=0x7f27ac00bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:07.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.034+0000 7f27b0ff9640 1 -- 172.21.15.16:0/3454607493 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac019070 con 0x7f27b410c180 2024-04-25T01:37:07.011 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.034+0000 7f27b0ff9640 1 -- 172.21.15.16:0/3454607493 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27ac00cba0 con 0x7f27b410c180 2024-04-25T01:37:07.012 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.034+0000 7f27b0ff9640 1 -- 172.21.15.16:0/3454607493 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0125e0 con 0x7f27b410c180 2024-04-25T01:37:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.034+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b41ae4d0 con 0x7f27b410c180 2024-04-25T01:37:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.034+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b41ae960 con 0x7f27b410c180 2024-04-25T01:37:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 Processor -- start 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 -- start start 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b88fd0 0x5642c7b893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b8e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 0x5642c7b90fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5642c7adbcb0 con 0x5642c7b899c0 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642c7a845b0 con 0x5642c7b88fd0 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27bcfca740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5642c7ad7c70 con 0x5642c7b8eb30 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 0x5642c7b90fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 0x5642c7b90fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:41694/0 (socket says 172.21.15.16:41694) 2024-04-25T01:37:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b88fd0 0x5642c7b893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.038+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1018693319 learned_addr learned my addr 172.21.15.16:0/1018693319 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b37fe640 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b8e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1018693319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b88fd0 msgr2=0x5642c7b893f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b88fd0 0x5642c7b893f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1018693319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 msgr2=0x5642c7b8e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b8e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1018693319 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c77f0210 con 0x5642c7b8eb30 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b37fe640 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.039+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 0x5642c7b90fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f27ac01a000 tx=0x7f27ac00cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a3c98d9f7bd0fc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.041+0000 7f27b0ff9640 1 -- 172.21.15.16:0/3454607493 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f27ac051020 con 0x7f27b410c180 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.041+0000 7f2799ffb640 1 -- 172.21.15.16:0/1018693319 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0d1680 con 0x5642c7b8eb30 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.041+0000 7f2799ffb640 1 -- 172.21.15.16:0/1018693319 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27ac0d1ec0 con 0x5642c7b8eb30 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.041+0000 7f2799ffb640 1 -- 172.21.15.16:0/1018693319 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0d3160 con 0x5642c7b8eb30 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27b0ff9640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f279c0a2d10 0x7f279c0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 -- 172.21.15.16:0/1018693319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 msgr2=0x5642c7b90fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 0x5642c7b90fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f27ac01a000 tx=0x7f27ac00cdd0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27b0ff9640 1 -- 172.21.15.16:0/3454607493 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f27ac0d10c0 con 0x7f27b410c180 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 -- 172.21.15.16:0/1018693319 shutdown_connections 2024-04-25T01:37:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b8eb30 0x5642c7b90fc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1018693319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b88fd0 0x5642c7b893f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 -- 172.21.15.16:0/1018693319 >> 172.21.15.16:0/1018693319 conn(0x5642c7b73630 msgr2=0x5642c7b73a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27b3fff640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f279c0a2d10 0x7f279c0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 -- 172.21.15.16:0/1018693319 shutdown_connections 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 -- 172.21.15.16:0/1018693319 wait complete. 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.042+0000 7f27bcfca740 1 -- start start 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 Processor -- start 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b3fff640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f279c0a2d10 0x7f279c0a51d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x5642c7b8a220 tx=0x7f27a4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b793a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b8eb30 0x5642c7b80920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5642c7b77ba0 con 0x5642c7b899c0 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642c7a97b50 con 0x5642c7b8eb30 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5642c7b77d20 con 0x5642c7b88fd0 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b798e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40546/0 (socket says 172.21.15.16:40546) 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3790990349 learned_addr learned my addr 172.21.15.16:0/3790990349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b793a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27bcfca740 10 client.0 _ll_register_callbacks cb 0x5642c7ad2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b3fff640 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b8eb30 0x5642c7b80920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3790990349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 msgr2=0x5642c7b793a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b793a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b8eb30 msgr2=0x5642c7b80920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b8eb30 0x5642c7b80920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.043+0000 7f27b2ffd640 1 -- 172.21.15.16:0/3790990349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c78bc620 con 0x5642c7b899c0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b798e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00bfd0 tx=0x7f27ac09d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eebff0613ba27e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f2792ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f2792ffd640 1 -- 172.21.15.16:0/3790990349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0b6060 con 0x5642c7b899c0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27ac0d1ec0 con 0x5642c7b899c0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f2792ffd640 1 -- 172.21.15.16:0/3790990349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f27ac09dd00 con 0x5642c7b899c0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f2792ffd640 1 -- 172.21.15.16:0/3790990349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f27ac0bd050 con 0x5642c7b899c0 2024-04-25T01:37:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f27bcfca740 10 client.15990 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5642c7b83be0 con 0x5642c7b899c0 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.044+0000 7f2792ffd640 1 -- 172.21.15.16:0/3790990349 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27ac0d1ec0 con 0x5642c7b899c0 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f2792ffd640 1 -- 172.21.15.16:0/3790990349 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f27ac026070 con 0x5642c7b899c0 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27bcfca740 10 client.15990 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27bcfca740 10 client.15990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27bcfca740 20 client.15990 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27bcfca740 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b84ff0 0x5642c7b87420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27bcfca740 4 client.15990 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642c7ae91f0 con 0x5642c7b84ff0 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27b37fe640 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b84ff0 0x5642c7b87420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f2792ffd640 10 client.15990 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.045+0000 7f27b37fe640 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b84ff0 0x5642c7b87420 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed090ed8a73ad95b server_cookie=195ea348c2e9e9c7 in_seq=0 out_seq=0 2024-04-25T01:37:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f2792ffd640 1 -- 172.21.15.16:0/3790990349 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642c7ae91f0 con 0x5642c7b84ff0 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f2792ffd640 10 client.15990 handle_command_reply: tid=0 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 client.15990 shutdown 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 msgr2=0x5642c7b798e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b798e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00bfd0 tx=0x7f27ac09d8f0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 shutdown_connections 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b84ff0 0x5642c7b87420 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f2792ffd640 0 client.15990 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b8eb30 0x5642c7b80920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b899c0 0x5642c7b798e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 --2- 172.21.15.16:0/3790990349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b793a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.046+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 >> 172.21.15.16:0/3790990349 conn(0x5642c7b73630 msgr2=0x5642c7b72830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.047+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 shutdown_connections 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.047+0000 7f27bcfca740 1 -- 172.21.15.16:0/3790990349 wait complete. 2024-04-25T01:37:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.047+0000 7f27bcfca740 20 client.15990 trim_cache size 0 max 16384 2024-04-25T01:37:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 Processor -- start 2024-04-25T01:37:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 -- start start 2024-04-25T01:37:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b8aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b808e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b8b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41b1720 con 0x5642c7b84ff0 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41aeff0 con 0x5642c7b899c0 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27bcfca740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41a87a0 con 0x5642c7b88fd0 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b8b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b808e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b8b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:37060/0 (socket says 172.21.15.16:37060) 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27b2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b808e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:41708/0 (socket says 172.21.15.16:41708) 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.084+0000 7f27b3fff640 1 -- 172.21.15.16:0/2205771937 learned_addr learned my addr 172.21.15.16:0/2205771937 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27b3fff640 1 -- 172.21.15.16:0/2205771937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 msgr2=0x5642c7b808e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27b3fff640 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b808e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27b3fff640 1 -- 172.21.15.16:0/2205771937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 msgr2=0x5642c7b8aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27b3fff640 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b8aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27b3fff640 1 -- 172.21.15.16:0/2205771937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c7b8a250 con 0x5642c7b899c0 2024-04-25T01:37:07.063 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27b3fff640 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b8b1e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f27a4033000 tx=0x7f27a403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1deb145ef160e0cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27997fa640 1 -- 172.21.15.16:0/2205771937 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4035070 con 0x5642c7b899c0 2024-04-25T01:37:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27997fa640 1 -- 172.21.15.16:0/2205771937 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27a4031bd0 con 0x5642c7b899c0 2024-04-25T01:37:07.064 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.085+0000 7f27997fa640 1 -- 172.21.15.16:0/2205771937 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a403cc70 con 0x5642c7b899c0 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.086+0000 7f27bcfca740 1 -- 172.21.15.16:0/2205771937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 msgr2=0x5642c7b8b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.086+0000 7f27bcfca740 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b8b1e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f27a4033000 tx=0x7f27a403f000 comp rx=0 tx=0).stop 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 -- 172.21.15.16:0/2205771937 shutdown_connections 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b8b1e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b808e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 --2- 172.21.15.16:0/2205771937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b8aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 -- 172.21.15.16:0/2205771937 >> 172.21.15.16:0/2205771937 conn(0x5642c7b73630 msgr2=0x5642c7b793a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 -- 172.21.15.16:0/2205771937 shutdown_connections 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 -- 172.21.15.16:0/2205771937 wait complete. 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 -- start start 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 Processor -- start 2024-04-25T01:37:07.069 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b801e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.087+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b80720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27bcfca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b7cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27bcfca740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41a8de0 con 0x5642c7b84ff0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27bcfca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41aa250 con 0x5642c7b899c0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27bcfca740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41a94b0 con 0x5642c7b88fd0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27bcfca740 10 client.0 _ll_register_callbacks cb 0x7f27ac099e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b7cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b7cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:37076/0 (socket says 172.21.15.16:37076) 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b3fff640 1 -- 172.21.15.16:0/1217994805 learned_addr learned my addr 172.21.15.16:0/1217994805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b80720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b37fe640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.070 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 msgr2=0x5642c7b7cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b7cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1217994805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 msgr2=0x5642c7b801e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b801e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.088+0000 7f27b2ffd640 1 -- 172.21.15.16:0/1217994805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c7b7d3f0 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f27b2ffd640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b80720 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f27ac09da30 tx=0x7f27ac0a44a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63b116206f8473c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f2792ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f2792ffd640 1 -- 172.21.15.16:0/1217994805 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac022070 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f2792ffd640 1 -- 172.21.15.16:0/1217994805 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f27ac0d2600 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27a4031bd0 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f2792ffd640 1 -- 172.21.15.16:0/1217994805 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f27ac04e050 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f27bcfca740 10 client.25627 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.089+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5642c7c15cb0 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.090+0000 7f2792ffd640 1 -- 172.21.15.16:0/1217994805 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27a4031bd0 con 0x5642c7b88fd0 2024-04-25T01:37:07.071 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f2792ffd640 1 -- 172.21.15.16:0/1217994805 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f27ac0997f0 con 0x5642c7b88fd0 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27bcfca740 10 client.25627 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27bcfca740 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27bcfca740 20 client.25627 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b8eb30 0x5642c7c184b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27bcfca740 4 client.25627 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5642c7ae3540 con 0x5642c7b8eb30 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.091+0000 7f27b37fe640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b8eb30 0x5642c7c184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.092+0000 7f27b37fe640 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b8eb30 0x5642c7c184b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fad7563bb36374c8 server_cookie=f3a3ad4d2f024b4e in_seq=0 out_seq=0 2024-04-25T01:37:07.072 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.092+0000 7f2792ffd640 10 client.25627 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.096+0000 7f2792ffd640 1 -- 172.21.15.16:0/1217994805 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5642c7ae3540 con 0x5642c7b8eb30 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.096+0000 7f2792ffd640 10 client.25627 handle_command_reply: tid=0 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.096+0000 7f27bcfca740 1 client.25627 shutdown 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 msgr2=0x5642c7b80720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b80720 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f27ac09da30 tx=0x7f27ac0a44a0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 shutdown_connections 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5642c7b8eb30 0x5642c7c184b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642c7b899c0 0x5642c7b7cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642c7b88fd0 0x5642c7b80720 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 --2- 172.21.15.16:0/1217994805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5642c7b84ff0 0x5642c7b801e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 >> 172.21.15.16:0/1217994805 conn(0x5642c7b73630 msgr2=0x5642c7b79380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:07.074 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f2792ffd640 0 client.25627 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:07.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 shutdown_connections 2024-04-25T01:37:07.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.097+0000 7f27bcfca740 1 -- 172.21.15.16:0/1217994805 wait complete. 2024-04-25T01:37:07.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.098+0000 7f27bcfca740 20 client.25627 trim_cache size 0 max 16384 2024-04-25T01:37:07.075 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:37:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:37:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:37:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:37:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f279c0a2d10 msgr2=0x7f279c0a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f279c0a2d10 0x7f279c0a51d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x5642c7b8a220 tx=0x7f27a4005850 comp rx=0 tx=0).stop 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 msgr2=0x7f27b419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 0x7f27b419de20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00b9e0 tx=0x7f27ac00bea0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 shutdown_connections 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f279c0a2d10 0x7f279c0a51d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27b410eab0 0x7f27b419a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27b410c180 0x7f27b419de20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 --2- 172.21.15.16:0/3454607493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27b4109850 0x7f27b419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.099+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 >> 172.21.15.16:0/3454607493 conn(0x7f27b406e7f0 msgr2=0x7f27b4108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.100+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 shutdown_connections 2024-04-25T01:37:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:07.100+0000 7f27ba1dc640 1 -- 172.21.15.16:0/3454607493 wait complete. 2024-04-25T01:37:07.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:07 smithi016 ceph-mon[29153]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 52 MiB/s wr, 937 op/s 2024-04-25T01:37:07.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:07 smithi038 ceph-mon[32841]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 52 MiB/s wr, 937 op/s 2024-04-25T01:37:07.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:07 smithi156 ceph-mon[32631]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 52 MiB/s wr, 937 op/s 2024-04-25T01:37:09.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:09 smithi016 ceph-mon[29153]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 63 MiB/s wr, 1.14k op/s 2024-04-25T01:37:09.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:09 smithi038 ceph-mon[32841]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 63 MiB/s wr, 1.14k op/s 2024-04-25T01:37:09.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:09 smithi156 ceph-mon[32631]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 63 MiB/s wr, 1.14k op/s 2024-04-25T01:37:11.799 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:11 smithi156 ceph-mon[32631]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 57 MiB/s wr, 1.03k op/s 2024-04-25T01:37:11.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:11 smithi016 ceph-mon[29153]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 57 MiB/s wr, 1.03k op/s 2024-04-25T01:37:11.962 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:11 smithi038 ceph-mon[32841]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 57 MiB/s wr, 1.03k op/s 2024-04-25T01:37:13.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:13 smithi016 ceph-mon[29153]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:37:13.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:13 smithi038 ceph-mon[32841]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:37:13.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:13 smithi156 ceph-mon[32631]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 58 MiB/s wr, 1.04k op/s 2024-04-25T01:37:15.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:15 smithi016 ceph-mon[29153]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:15.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:15 smithi038 ceph-mon[32841]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:15.997 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:15 smithi156 ceph-mon[32631]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:17.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:17 smithi016 ceph-mon[29153]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 52 MiB/s wr, 924 op/s 2024-04-25T01:37:17.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:17 smithi038 ceph-mon[32841]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 52 MiB/s wr, 924 op/s 2024-04-25T01:37:17.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:17 smithi156 ceph-mon[32631]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 52 MiB/s wr, 924 op/s 2024-04-25T01:37:19.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:19 smithi016 ceph-mon[29153]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 63 MiB/s wr, 1.13k op/s 2024-04-25T01:37:19.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:19 smithi038 ceph-mon[32841]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 63 MiB/s wr, 1.13k op/s 2024-04-25T01:37:19.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:19 smithi156 ceph-mon[32631]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 63 MiB/s wr, 1.13k op/s 2024-04-25T01:37:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.267+0000 7faa3115a640 1 -- 172.21.15.16:0/3609806907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c1093b0 msgr2=0x7faa2c072790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.267+0000 7faa3115a640 1 --2- 172.21.15.16:0/3609806907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c1093b0 0x7faa2c072790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009860 tx=0x7faa1c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.269+0000 7faa3115a640 1 -- 172.21.15.16:0/3609806907 shutdown_connections 2024-04-25T01:37:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.269+0000 7faa3115a640 1 --2- 172.21.15.16:0/3609806907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 0x7faa2c111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.269+0000 7faa3115a640 1 --2- 172.21.15.16:0/3609806907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c072d60 0x7faa2c0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.269+0000 7faa3115a640 1 --2- 172.21.15.16:0/3609806907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c1093b0 0x7faa2c072790 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.269+0000 7faa3115a640 1 -- 172.21.15.16:0/3609806907 >> 172.21.15.16:0/3609806907 conn(0x7faa2c06e5e0 msgr2=0x7faa2c070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.274+0000 7faa3115a640 1 -- 172.21.15.16:0/3609806907 shutdown_connections 2024-04-25T01:37:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.274+0000 7faa3115a640 1 -- 172.21.15.16:0/3609806907 wait complete. 2024-04-25T01:37:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.275+0000 7faa3115a640 1 Processor -- start 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 -- start start 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c072d60 0x7faa2c1a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 0x7faa2c1a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 0x7faa2c1a9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c117bd0 con 0x7faa2c1093b0 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c117a50 con 0x7faa2c10fad0 2024-04-25T01:37:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa3115a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c117d50 con 0x7faa2c072d60 2024-04-25T01:37:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c072d60 0x7faa2c1a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa2a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 0x7faa2c1a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa2a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 0x7faa2c1a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51820/0 (socket says 172.21.15.16:51820) 2024-04-25T01:37:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa2a59c640 1 -- 172.21.15.16:0/1581308372 learned_addr learned my addr 172.21.15.16:0/1581308372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.276+0000 7faa2b59e640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 0x7faa2c1a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2a59c640 1 -- 172.21.15.16:0/1581308372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c072d60 msgr2=0x7faa2c1a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2a59c640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c072d60 0x7faa2c1a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2a59c640 1 -- 172.21.15.16:0/1581308372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 msgr2=0x7faa2c1a9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2a59c640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 0x7faa2c1a9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2a59c640 1 -- 172.21.15.16:0/1581308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c1a9ef0 con 0x7faa2c1093b0 2024-04-25T01:37:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2b59e640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 0x7faa2c1a9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.277+0000 7faa2a59c640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 0x7faa2c1a2810 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faa2000b4a0 tx=0x7faa2000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.278+0000 7faa13fff640 1 -- 172.21.15.16:0/1581308372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20014070 con 0x7faa2c1093b0 2024-04-25T01:37:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.278+0000 7faa13fff640 1 -- 172.21.15.16:0/1581308372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa20010ac0 con 0x7faa2c1093b0 2024-04-25T01:37:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.278+0000 7faa13fff640 1 -- 172.21.15.16:0/1581308372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20017c30 con 0x7faa2c1093b0 2024-04-25T01:37:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.278+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2c1aa1e0 con 0x7faa2c1093b0 2024-04-25T01:37:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.278+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2c1aa760 con 0x7faa2c1093b0 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.281+0000 7faa33e62740 1 Processor -- start 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 -- start start 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa13fff640 1 -- 172.21.15.16:0/1581308372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7faa20017070 con 0x7faa2c1093b0 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8001cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f3f7f57cb0 con 0x55f3f8002330 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f3f7f005b0 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa33e62740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f3f7f53c70 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8001cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45688/0 (socket says 172.21.15.16:45688) 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8001cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:55998/0 (socket says 172.21.15.16:55998) 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2b59e640 1 -- 172.21.15.16:0/2693090613 learned_addr learned my addr 172.21.15.16:0/2693090613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2a59c640 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 -- 172.21.15.16:0/2693090613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 msgr2=0x55f3f7ffcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 -- 172.21.15.16:0/2693090613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 msgr2=0x55f3f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2ad9d640 1 -- 172.21.15.16:0/2693090613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f7c6c210 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.282+0000 7faa2a59c640 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8001cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009830 tx=0x7faa1c0306e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e2b4c25007caaf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa13fff640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa140a2bc0 0x7faa140a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa117fa640 1 -- 172.21.15.16:0/2693090613 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c030c70 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa117fa640 1 -- 172.21.15.16:0/2693090613 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c041070 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa117fa640 1 -- 172.21.15.16:0/2693090613 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c004c80 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa13fff640 1 -- 172.21.15.16:0/1581308372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa200ce6a0 con 0x7faa2c1093b0 2024-04-25T01:37:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.283+0000 7faa2b59e640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa140a2bc0 0x7faa140a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 -- 172.21.15.16:0/2693090613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 msgr2=0x55f3f8001cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa2b59e640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa140a2bc0 0x7faa140a5080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55f3f7ffda20 tx=0x7faa24010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8001cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009830 tx=0x7faa1c0306e0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 -- 172.21.15.16:0/2693090613 shutdown_connections 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8001cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 --2- 172.21.15.16:0/2693090613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 -- 172.21.15.16:0/2693090613 >> 172.21.15.16:0/2693090613 conn(0x55f3f7ff7e40 msgr2=0x55f3f7ff8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 -- 172.21.15.16:0/2693090613 shutdown_connections 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.284+0000 7faa33e62740 1 -- 172.21.15.16:0/2693090613 wait complete. 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.285+0000 7faa33e62740 1 -- start start 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.285+0000 7faa33e62740 1 Processor -- start 2024-04-25T01:37:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.285+0000 7faa33e62740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffc7d0 0x55f3f8006c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.285+0000 7faa33e62740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8007140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.285+0000 7faa33e62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f800e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f800e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa33e62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f3f8005280 con 0x55f3f8002330 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa33e62740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f3f7f13b50 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa33e62740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f3f8005400 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f800e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51828/0 (socket says 172.21.15.16:51828) 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2b59e640 1 -- 172.21.15.16:0/149182637 learned_addr learned my addr 172.21.15.16:0/149182637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffc7d0 0x55f3f8006c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2a59c640 1 -- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffc7d0 msgr2=0x55f3f8006c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2a59c640 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffc7d0 0x55f3f8006c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2a59c640 1 -- 172.21.15.16:0/149182637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 msgr2=0x55f3f800e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2a59c640 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f800e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.286+0000 7faa2a59c640 1 -- 172.21.15.16:0/149182637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f7d38620 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa33e62740 10 client.0 _ll_register_callbacks cb 0x55f3f7f4eca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c041070 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa2a59c640 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8007140 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faa200cc1d0 tx=0x7faa2009ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e812626a7db0b278 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa0eff5640 10 client.25639 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa0eff5640 1 -- 172.21.15.16:0/149182637 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2001e070 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa0eff5640 1 -- 172.21.15.16:0/149182637 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa20097600 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.287+0000 7faa0eff5640 1 -- 172.21.15.16:0/149182637 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faa2009af90 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa33e62740 10 client.25639 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55f3f80113d0 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa0eff5640 1 -- 172.21.15.16:0/149182637 <== mon.1 v2:172.21.15.156:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faa20010c70 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa33e62740 10 client.25639 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa33e62740 10 client.25639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa33e62740 20 client.25639 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.288+0000 7faa33e62740 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8012a70 0x55f3f8014f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.289+0000 7faa33e62740 4 client.25639 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.289+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3f7f651f0 con 0x55f3f8012a70 2024-04-25T01:37:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.289+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8012a70 0x55f3f8014f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.289+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8012a70 0x55f3f8014f00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3c2abe5b43f7a21 server_cookie=8c4337ebf10e49a3 in_seq=0 out_seq=0 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.290+0000 7faa0eff5640 10 client.25639 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.290+0000 7faa0eff5640 1 -- 172.21.15.16:0/149182637 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3f7f651f0 con 0x55f3f8012a70 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa0eff5640 10 client.25639 handle_command_reply: tid=0 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa33e62740 1 client.25639 shutdown 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 msgr2=0x55f3f8007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa33e62740 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8007140 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faa200cc1d0 tx=0x7faa2009ad60 comp rx=0 tx=0).stop 2024-04-25T01:37:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 shutdown_connections 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa33e62740 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f8002330 0x55f3f800e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.291+0000 7faa33e62740 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f8007140 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa33e62740 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8012a70 0x55f3f8014f00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa33e62740 1 --2- 172.21.15.16:0/149182637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f7ffc7d0 0x55f3f8006c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 >> 172.21.15.16:0/149182637 conn(0x55f3f7ff7e40 msgr2=0x55f3f7fee830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa0eff5640 0 client.25639 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 shutdown_connections 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa33e62740 1 -- 172.21.15.16:0/149182637 wait complete. 2024-04-25T01:37:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.292+0000 7faa33e62740 20 client.25639 trim_cache size 0 max 16384 2024-04-25T01:37:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.324+0000 7faa33e62740 1 Processor -- start 2024-04-25T01:37:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.324+0000 7faa33e62740 1 -- start start 2024-04-25T01:37:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa33e62740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa33e62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 0x55f3f7ffeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffe5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45706/0 (socket says 172.21.15.16:45706) 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa2a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 0x55f3f7ffeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa2a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 0x55f3f7ffeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51844/0 (socket says 172.21.15.16:51844) 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa2ad9d640 1 -- 172.21.15.16:0/984360068 learned_addr learned my addr 172.21.15.16:0/984360068 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa33e62740 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.325+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1b1a80 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa2b59e640 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1af350 con 0x55f3f8002330 2024-04-25T01:37:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1ad1a0 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa2a59c640 1 -- 172.21.15.16:0/984360068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 msgr2=0x55f3f7ffe5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa2a59c640 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffe5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa2a59c640 1 -- 172.21.15.16:0/984360068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 msgr2=0x55f3f800e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa2a59c640 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.326+0000 7faa2a59c640 1 -- 172.21.15.16:0/984360068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f8001880 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.327+0000 7faa2a59c640 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 0x55f3f7ffeb20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7faa20097960 tx=0x7faa200bb000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a0ba8bd3ed536c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.327+0000 7faa10ff9640 1 -- 172.21.15.16:0/984360068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20099290 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.327+0000 7faa10ff9640 1 -- 172.21.15.16:0/984360068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c041070 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.327+0000 7faa10ff9640 1 -- 172.21.15.16:0/984360068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa200a3920 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 msgr2=0x55f3f7ffeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 0x55f3f7ffeb20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7faa20097960 tx=0x7faa200bb000 comp rx=0 tx=0).stop 2024-04-25T01:37:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 shutdown_connections 2024-04-25T01:37:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffd1c0 0x55f3f7ffeb20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 --2- 172.21.15.16:0/984360068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffc7d0 0x55f3f7ffe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 >> 172.21.15.16:0/984360068 conn(0x55f3f7ff7e40 msgr2=0x55f3f7ff8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 shutdown_connections 2024-04-25T01:37:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 -- 172.21.15.16:0/984360068 wait complete. 2024-04-25T01:37:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.329+0000 7faa33e62740 1 -- start start 2024-04-25T01:37:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 Processor -- start 2024-04-25T01:37:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 0x55f3f800bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f800c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1ad810 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1aec80 con 0x55f3f8002330 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1adee0 con 0x55f3f7ffd1c0 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa33e62740 10 client.0 _ll_register_callbacks cb 0x7faa200175f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 0x55f3f800bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa2ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 0x55f3f800bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51858/0 (socket says 172.21.15.16:51858) 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.330+0000 7faa2ad9d640 1 -- 172.21.15.16:0/1725358989 learned_addr learned my addr 172.21.15.16:0/1725358989 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2a59c640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f800c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2b59e640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2ad9d640 1 -- 172.21.15.16:0/1725358989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 msgr2=0x55f3f800c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f800c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2ad9d640 1 -- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 msgr2=0x55f3f800d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2ad9d640 1 -- 172.21.15.16:0/1725358989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f800d690 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa2ad9d640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 0x55f3f800bbc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x55f3f7ffdc30 tx=0x7faa1c030bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5154334b98d59da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:37:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.331+0000 7faa0eff5640 1 -- 172.21.15.16:0/1725358989 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c005240 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.332+0000 7faa0eff5640 1 -- 172.21.15.16:0/1725358989 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa1c0053e0 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.332+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c041070 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.332+0000 7faa0eff5640 1 -- 172.21.15.16:0/1725358989 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faa1c0055e0 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.332+0000 7faa33e62740 10 client.16008 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.332+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f3f8079300 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.332+0000 7faa0eff5640 1 -- 172.21.15.16:0/1725358989 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c041070 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa0eff5640 1 -- 172.21.15.16:0/1725358989 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faa1c04e040 con 0x55f3f7ffc7d0 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa33e62740 10 client.16008 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa33e62740 10 client.16008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa33e62740 20 client.16008 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa33e62740 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8079930 0x55f3f807bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa33e62740 4 client.16008 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:37:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f3f7f5f540 con 0x55f3f8079930 2024-04-25T01:37:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.333+0000 7faa2a59c640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8079930 0x55f3f807bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.334+0000 7faa2a59c640 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8079930 0x55f3f807bd20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b6a81c86d4e26b server_cookie=44bf8377dfa37428 in_seq=0 out_seq=0 2024-04-25T01:37:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.334+0000 7faa0eff5640 10 client.16008 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.342+0000 7faa0eff5640 1 -- 172.21.15.16:0/1725358989 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f3f7f5f540 con 0x55f3f8079930 2024-04-25T01:37:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.342+0000 7faa0eff5640 10 client.16008 handle_command_reply: tid=0 2024-04-25T01:37:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.343+0000 7faa33e62740 1 client.16008 shutdown 2024-04-25T01:37:20.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.345+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 msgr2=0x55f3f800bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.322 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.345+0000 7faa33e62740 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 0x55f3f800bbc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x55f3f7ffdc30 tx=0x7faa1c030bb0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 shutdown_connections 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55f3f8079930 0x55f3f807bd20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3f8002330 0x55f3f800d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa0eff5640 0 client.16008 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3f7ffd1c0 0x55f3f800c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 --2- 172.21.15.16:0/1725358989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f3f7ffc7d0 0x55f3f800bbc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 >> 172.21.15.16:0/1725358989 conn(0x55f3f7ff7e40 msgr2=0x55f3f8006c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 shutdown_connections 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.346+0000 7faa33e62740 1 -- 172.21.15.16:0/1725358989 wait complete. 2024-04-25T01:37:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.347+0000 7faa33e62740 20 client.16008 trim_cache size 0 max 16384 2024-04-25T01:37:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:37:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:37:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:37:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:37:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa140a2bc0 msgr2=0x7faa140a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa140a2bc0 0x7faa140a5080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55f3f7ffda20 tx=0x7faa24010040 comp rx=0 tx=0).stop 2024-04-25T01:37:20.326 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 msgr2=0x7faa2c1a2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 0x7faa2c1a2810 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faa2000b4a0 tx=0x7faa2000b960 comp rx=0 tx=0).stop 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 shutdown_connections 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7faa140a2bc0 0x7faa140a5080 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.348+0000 7faa3115a640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa2c10fad0 0x7faa2c1a9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.349+0000 7faa3115a640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2c1093b0 0x7faa2c1a2810 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.349+0000 7faa3115a640 1 --2- 172.21.15.16:0/1581308372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa2c072d60 0x7faa2c1a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.349+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 >> 172.21.15.16:0/1581308372 conn(0x7faa2c06e5e0 msgr2=0x7faa2c10b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.350+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 shutdown_connections 2024-04-25T01:37:20.327 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:20.350+0000 7faa3115a640 1 -- 172.21.15.16:0/1581308372 wait complete. 2024-04-25T01:37:21.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:21 smithi016 ceph-mon[29153]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-04-25T01:37:21.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:21 smithi038 ceph-mon[32841]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-04-25T01:37:21.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:21 smithi156 ceph-mon[32631]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 57 MiB/s wr, 1.02k op/s 2024-04-25T01:37:23.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:23 smithi016 ceph-mon[29153]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:37:23.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:23 smithi038 ceph-mon[32841]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:37:23.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:23 smithi156 ceph-mon[32631]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-04-25T01:37:25.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:25 smithi016 ceph-mon[29153]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:25.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:25 smithi038 ceph-mon[32841]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:25.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:25 smithi156 ceph-mon[32631]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:27.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:27 smithi016 ceph-mon[29153]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 52 MiB/s wr, 925 op/s 2024-04-25T01:37:27.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:27 smithi038 ceph-mon[32841]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 52 MiB/s wr, 925 op/s 2024-04-25T01:37:27.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:27 smithi156 ceph-mon[32631]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 52 MiB/s wr, 925 op/s 2024-04-25T01:37:29.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:29 smithi016 ceph-mon[29153]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:29.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:29 smithi038 ceph-mon[32841]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:29.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:29 smithi156 ceph-mon[32631]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 63 MiB/s wr, 1.12k op/s 2024-04-25T01:37:31.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:31 smithi016 ceph-mon[29153]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 57 MiB/s wr, 1.01k op/s 2024-04-25T01:37:31.966 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:31 smithi038 ceph-mon[32841]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 57 MiB/s wr, 1.01k op/s 2024-04-25T01:37:31.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:31 smithi156 ceph-mon[32631]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 57 MiB/s wr, 1.01k op/s 2024-04-25T01:37:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.514+0000 7fab31ffb640 1 -- 172.21.15.16:0/3936608421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24031880 con 0x7fab34100040 2024-04-25T01:37:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 -- 172.21.15.16:0/3936608421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34100040 msgr2=0x7fab34102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 --2- 172.21.15.16:0/3936608421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34100040 0x7fab34102470 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fab24009900 tx=0x7fab2402a410 comp rx=0 tx=0).stop 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 -- 172.21.15.16:0/3936608421 shutdown_connections 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 --2- 172.21.15.16:0/3936608421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab34105360 0x7fab341057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 --2- 172.21.15.16:0/3936608421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab341029b0 0x7fab34104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 --2- 172.21.15.16:0/3936608421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34100040 0x7fab34102470 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.515+0000 7fab399dc640 1 -- 172.21.15.16:0/3936608421 >> 172.21.15.16:0/3936608421 conn(0x7fab340fbc40 msgr2=0x7fab340fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.516+0000 7fab399dc640 1 -- 172.21.15.16:0/3936608421 shutdown_connections 2024-04-25T01:37:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.516+0000 7fab399dc640 1 -- 172.21.15.16:0/3936608421 wait complete. 2024-04-25T01:37:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 Processor -- start 2024-04-25T01:37:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 -- start start 2024-04-25T01:37:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab34100040 0x7fab3419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab341029b0 0x7fab3419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34105360 0x7fab341a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab34113940 con 0x7fab34105360 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fab341137c0 con 0x7fab34100040 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.517+0000 7fab399dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fab34113ac0 con 0x7fab341029b0 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34105360 0x7fab341a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34105360 0x7fab341a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51268/0 (socket says 172.21.15.16:51268) 2024-04-25T01:37:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab337fe640 1 -- 172.21.15.16:0/2610697708 learned_addr learned my addr 172.21.15.16:0/2610697708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab32ffd640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab34100040 0x7fab3419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab327fc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab341029b0 0x7fab3419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab32ffd640 1 -- 172.21.15.16:0/2610697708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab341029b0 msgr2=0x7fab3419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab32ffd640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab341029b0 0x7fab3419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab32ffd640 1 -- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34105360 msgr2=0x7fab341a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab32ffd640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34105360 0x7fab341a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.518+0000 7fab32ffd640 1 -- 172.21.15.16:0/2610697708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab341a5b20 con 0x7fab34100040 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.519+0000 7fab32ffd640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab34100040 0x7fab3419df00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab2402a930 tx=0x7fab24003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.519+0000 7fab389da640 1 -- 172.21.15.16:0/2610697708 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2403f070 con 0x7fab34100040 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.520+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab341a5db0 con 0x7fab34100040 2024-04-25T01:37:33.497 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.520+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab341a61c0 con 0x7fab34100040 2024-04-25T01:37:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.520+0000 7fab389da640 1 -- 172.21.15.16:0/2610697708 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab24031c40 con 0x7fab34100040 2024-04-25T01:37:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.521+0000 7fab389da640 1 -- 172.21.15.16:0/2610697708 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24035030 con 0x7fab34100040 2024-04-25T01:37:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.523+0000 7fab389da640 1 -- 172.21.15.16:0/2610697708 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fab24042070 con 0x7fab34100040 2024-04-25T01:37:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 Processor -- start 2024-04-25T01:37:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 -- start start 2024-04-25T01:37:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39ad1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad1840 0x55be39ad6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ad8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39ad1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39ad1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60966/0 (socket says 172.21.15.16:60966) 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab337fe640 1 -- 172.21.15.16:0/3259529534 learned_addr learned my addr 172.21.15.16:0/3259529534 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be39a34cb0 con 0x55be39ad69b0 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55be399dd5b0 con 0x55be39ad0e50 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55be39a30c70 con 0x55be39ad1840 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.524+0000 7fab32ffd640 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad1840 0x55be39ad6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab327fc640 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ad8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab337fe640 1 -- 172.21.15.16:0/3259529534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad1840 msgr2=0x55be39ad6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab337fe640 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad1840 0x55be39ad6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab337fe640 1 -- 172.21.15.16:0/3259529534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 msgr2=0x55be39ad8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab337fe640 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ad8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab337fe640 1 -- 172.21.15.16:0/3259529534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be39749210 con 0x55be39ad0e50 2024-04-25T01:37:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab389da640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fab1c0a2cf0 0x7fab1c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.525+0000 7fab337fe640 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39ad1270 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab2c011570 tx=0x7fab2c011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7eaca602f715fc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.526+0000 7fab32ffd640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fab1c0a2cf0 0x7fab1c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.526+0000 7fab19ffb640 1 -- 172.21.15.16:0/3259529534 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c004b20 con 0x55be39ad0e50 2024-04-25T01:37:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.526+0000 7fab19ffb640 1 -- 172.21.15.16:0/3259529534 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab2c004cc0 con 0x55be39ad0e50 2024-04-25T01:37:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.526+0000 7fab19ffb640 1 -- 172.21.15.16:0/3259529534 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c005480 con 0x55be39ad0e50 2024-04-25T01:37:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.526+0000 7fab389da640 1 -- 172.21.15.16:0/2610697708 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fab240ef860 con 0x7fab34100040 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 msgr2=0x55be39ad1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab32ffd640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fab1c0a2cf0 0x7fab1c0a51b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fab2402d020 tx=0x7fab240f5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39ad1270 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab2c011570 tx=0x7fab2c011a30 comp rx=0 tx=0).stop 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 shutdown_connections 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ad8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad1840 0x55be39ad6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/3259529534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39ad1270 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 >> 172.21.15.16:0/3259529534 conn(0x55be39acc4c0 msgr2=0x55be39acc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 shutdown_connections 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 -- 172.21.15.16:0/3259529534 wait complete. 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.527+0000 7fab3c7c7740 1 -- start start 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.528+0000 7fab3c7c7740 1 Processor -- start 2024-04-25T01:37:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.528+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39adb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.528+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad1840 0x55be39adb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.528+0000 7fab327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad1840 0x55be39adb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.528+0000 7fab337fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39adb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.528+0000 7fab337fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39adb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35918/0 (socket says 172.21.15.16:35918) 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad1840 0x55be39adb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60982/0 (socket says 172.21.15.16:60982) 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ae2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab337fe640 1 -- 172.21.15.16:0/924787995 learned_addr learned my addr 172.21.15.16:0/924787995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be39ad9960 con 0x55be39ad69b0 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55be399f0b50 con 0x55be39ad1840 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55be39ad9ae0 con 0x55be39ad0e50 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab3c7c7740 10 client.0 _ll_register_callbacks cb 0x55be39a2bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab337fe640 1 -- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad1840 msgr2=0x55be39adb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab337fe640 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad1840 0x55be39adb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab337fe640 1 -- 172.21.15.16:0/924787995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 msgr2=0x55be39ae2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab337fe640 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ae2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.529+0000 7fab337fe640 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be39815620 con 0x55be39ad0e50 2024-04-25T01:37:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.530+0000 7fab337fe640 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39adb0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab2c01b040 tx=0x7fab2c009d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af155f8e8c1f453 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.530+0000 7fab12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:37:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.530+0000 7fab12ffd640 1 -- 172.21.15.16:0/924787995 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c028070 con 0x55be39ad0e50 2024-04-25T01:37:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.530+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab2c004cc0 con 0x55be39ad0e50 2024-04-25T01:37:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.530+0000 7fab12ffd640 1 -- 172.21.15.16:0/924787995 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fab2c004ac0 con 0x55be39ad0e50 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab12ffd640 1 -- 172.21.15.16:0/924787995 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fab2c034050 con 0x55be39ad0e50 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 10 client.25657 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55be39ae5a30 con 0x55be39ad0e50 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab12ffd640 1 -- 172.21.15.16:0/924787995 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab2c004cc0 con 0x55be39ad0e50 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab12ffd640 1 -- 172.21.15.16:0/924787995 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fab2c024900 con 0x55be39ad0e50 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 10 client.25657 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 10 client.25657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 20 client.25657 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39ae74c0 0x55be39ae9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 4 client.25657 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.531+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be39a421f0 con 0x55be39ae74c0 2024-04-25T01:37:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.532+0000 7fab327fc640 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39ae74c0 0x55be39ae9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.533+0000 7fab327fc640 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39ae74c0 0x55be39ae9900 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9ada158c5c49e80 server_cookie=b278dd3770261ccd in_seq=0 out_seq=0 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.533+0000 7fab12ffd640 10 client.25657 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab12ffd640 1 -- 172.21.15.16:0/924787995 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be39a421f0 con 0x55be39ae74c0 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab12ffd640 10 client.25657 handle_command_reply: tid=0 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 client.25657 shutdown 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 msgr2=0x55be39adb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39adb0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab2c01b040 tx=0x7fab2c009d30 comp rx=0 tx=0).stop 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 shutdown_connections 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad69b0 0x55be39ae2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39ae74c0 0x55be39ae9900 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad1840 0x55be39adb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/924787995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39adb0a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 >> 172.21.15.16:0/924787995 conn(0x55be39acc4c0 msgr2=0x55be39acb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab12ffd640 0 client.25657 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 shutdown_connections 2024-04-25T01:37:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 1 -- 172.21.15.16:0/924787995 wait complete. 2024-04-25T01:37:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.534+0000 7fab3c7c7740 20 client.25657 trim_cache size 0 max 16384 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.567+0000 7fab3c7c7740 1 Processor -- start 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.567+0000 7fab3c7c7740 1 -- start start 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39aef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39aef750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad69b0 0x55be39ae25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab3c7c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab20010ec0 con 0x55be39ad1840 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab3c7c7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fab2000e790 con 0x55be39ad0e50 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab3c7c7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fab2000c5e0 con 0x55be39ad69b0 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39aef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab327fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39aef750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.546 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39aef210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:60992/0 (socket says 172.21.15.16:60992) 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 -- 172.21.15.16:0/2509266906 learned_addr learned my addr 172.21.15.16:0/2509266906 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad69b0 0x55be39ae25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 -- 172.21.15.16:0/2509266906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad69b0 msgr2=0x55be39ae25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad69b0 0x55be39ae25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 -- 172.21.15.16:0/2509266906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 msgr2=0x55be39aef750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39aef750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab337fe640 1 -- 172.21.15.16:0/2509266906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be39adbe60 con 0x55be39ad0e50 2024-04-25T01:37:33.547 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.568+0000 7fab327fc640 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39aef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:33.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.569+0000 7fab337fe640 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39aef210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fab2c011570 tx=0x7fab2c014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416c22343ec3c8bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.569+0000 7fab197fa640 1 -- 172.21.15.16:0/2509266906 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c028070 con 0x55be39ad0e50 2024-04-25T01:37:33.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.569+0000 7fab197fa640 1 -- 172.21.15.16:0/2509266906 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab2c004c80 con 0x55be39ad0e50 2024-04-25T01:37:33.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.569+0000 7fab197fa640 1 -- 172.21.15.16:0/2509266906 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c019e20 con 0x55be39ad0e50 2024-04-25T01:37:33.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 -- 172.21.15.16:0/2509266906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 msgr2=0x55be39aef210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39aef210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fab2c011570 tx=0x7fab2c014040 comp rx=0 tx=0).stop 2024-04-25T01:37:33.549 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 -- 172.21.15.16:0/2509266906 shutdown_connections 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad69b0 0x55be39ae25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39aef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/2509266906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad0e50 0x55be39aef210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.570+0000 7fab3c7c7740 1 -- 172.21.15.16:0/2509266906 >> 172.21.15.16:0/2509266906 conn(0x55be39acc4c0 msgr2=0x55be39ad2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.571+0000 7fab3c7c7740 1 -- 172.21.15.16:0/2509266906 shutdown_connections 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.571+0000 7fab3c7c7740 1 -- 172.21.15.16:0/2509266906 wait complete. 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.571+0000 7fab3c7c7740 1 -- start start 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.571+0000 7fab3c7c7740 1 Processor -- start 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39acb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39ae1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad69b0 0x55be39ae1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab2000cc50 con 0x55be39ad1840 2024-04-25T01:37:33.550 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab337fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39acb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fab2000e0c0 con 0x55be39ad69b0 2024-04-25T01:37:33.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab337fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39acb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35932/0 (socket says 172.21.15.16:35932) 2024-04-25T01:37:33.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab337fe640 1 -- 172.21.15.16:0/1486545960 learned_addr learned my addr 172.21.15.16:0/1486545960 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:33.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fab2000d320 con 0x55be39ad0e50 2024-04-25T01:37:33.551 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab3c7c7740 10 client.0 _ll_register_callbacks cb 0x7fab2c011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:33.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.572+0000 7fab337fe640 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad69b0 0x55be39ae1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab337fe640 1 -- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad69b0 msgr2=0x55be39ae1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab337fe640 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad69b0 0x55be39ae1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab337fe640 1 -- 172.21.15.16:0/1486545960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 msgr2=0x55be39ae1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.552 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab337fe640 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39ae1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab337fe640 1 -- 172.21.15.16:0/1486545960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be39ae2160 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab337fe640 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39acb500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fab2c011a70 tx=0x7fab2c00fb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3533de8e6be1fa90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab12ffd640 1 -- 172.21.15.16:0/1486545960 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c028050 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab12ffd640 1 -- 172.21.15.16:0/1486545960 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fab2c019ba0 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.573+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab2c004c80 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab12ffd640 1 -- 172.21.15.16:0/1486545960 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fab2c019da0 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 10 client.25663 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55be39b4c3e0 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab12ffd640 1 -- 172.21.15.16:0/1486545960 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab2c004c80 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab12ffd640 1 -- 172.21.15.16:0/1486545960 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fab2c005630 con 0x55be39ad0e50 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 10 client.25663 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:33.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 20 client.25663 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39b4cd60 0x55be39b4f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 4 client.25663 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:37:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.574+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be39a3c540 con 0x55be39b4cd60 2024-04-25T01:37:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.575+0000 7fab327fc640 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39b4cd60 0x55be39b4f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.575+0000 7fab12ffd640 10 client.25663 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:33.554 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.576+0000 7fab327fc640 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39b4cd60 0x55be39b4f180 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95ce934c83191ed2 server_cookie=b97ac124ee548f24 in_seq=0 out_seq=0 2024-04-25T01:37:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.584+0000 7fab12ffd640 1 -- 172.21.15.16:0/1486545960 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55be39a3c540 con 0x55be39b4cd60 2024-04-25T01:37:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.584+0000 7fab12ffd640 10 client.25663 handle_command_reply: tid=0 2024-04-25T01:37:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.585+0000 7fab3c7c7740 1 client.25663 shutdown 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.585+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 msgr2=0x55be39acb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.585+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39acb500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fab2c011a70 tx=0x7fab2c00fb70 comp rx=0 tx=0).stop 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 shutdown_connections 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55be39b4cd60 0x55be39b4f180 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be39ad69b0 0x55be39ae1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be39ad1840 0x55be39ae1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 --2- 172.21.15.16:0/1486545960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be39ad0e50 0x55be39acb500 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.563 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab12ffd640 0 client.25663 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 >> 172.21.15.16:0/1486545960 conn(0x55be39acc4c0 msgr2=0x55be39acf550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.586+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 shutdown_connections 2024-04-25T01:37:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.587+0000 7fab3c7c7740 1 -- 172.21.15.16:0/1486545960 wait complete. 2024-04-25T01:37:33.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.587+0000 7fab3c7c7740 20 client.25663 trim_cache size 0 max 16384 2024-04-25T01:37:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:37:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:37:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:37:33.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fab1c0a2cf0 msgr2=0x7fab1c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fab1c0a2cf0 0x7fab1c0a51b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fab2402d020 tx=0x7fab240f5530 comp rx=0 tx=0).stop 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab34100040 msgr2=0x7fab3419df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab34100040 0x7fab3419df00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab2402a930 tx=0x7fab24003c10 comp rx=0 tx=0).stop 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 shutdown_connections 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fab1c0a2cf0 0x7fab1c0a51b0 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fab2402d020 tx=0x7fab240f5530 comp rx=0 tx=0).stop 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab34105360 0x7fab341a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab341029b0 0x7fab3419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 --2- 172.21.15.16:0/2610697708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab34100040 0x7fab3419df00 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab2402a930 tx=0x7fab24003c10 comp rx=0 tx=0).stop 2024-04-25T01:37:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.589+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 >> 172.21.15.16:0/2610697708 conn(0x7fab340fbc40 msgr2=0x7fab34103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.590+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 shutdown_connections 2024-04-25T01:37:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:33.590+0000 7fab399dc640 1 -- 172.21.15.16:0/2610697708 wait complete. 2024-04-25T01:37:33.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:33 smithi016 ceph-mon[29153]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 58 MiB/s wr, 1.02k op/s 2024-04-25T01:37:33.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:33 smithi038 ceph-mon[32841]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 58 MiB/s wr, 1.02k op/s 2024-04-25T01:37:33.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:33 smithi156 ceph-mon[32631]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 58 MiB/s wr, 1.02k op/s 2024-04-25T01:37:35.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:35 smithi016 ceph-mon[29153]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 63 MiB/s wr, 1.11k op/s 2024-04-25T01:37:35.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:35 smithi038 ceph-mon[32841]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 63 MiB/s wr, 1.11k op/s 2024-04-25T01:37:35.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:35 smithi156 ceph-mon[32631]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 63 MiB/s wr, 1.11k op/s 2024-04-25T01:37:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:37 smithi016 ceph-mon[29153]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 52 MiB/s wr, 914 op/s 2024-04-25T01:37:37.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:37 smithi038 ceph-mon[32841]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 52 MiB/s wr, 914 op/s 2024-04-25T01:37:37.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:37 smithi156 ceph-mon[32631]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 52 MiB/s wr, 914 op/s 2024-04-25T01:37:39.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:39 smithi016 ceph-mon[29153]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 63 MiB/s wr, 1.10k op/s 2024-04-25T01:37:39.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:39 smithi038 ceph-mon[32841]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 63 MiB/s wr, 1.10k op/s 2024-04-25T01:37:39.998 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:39 smithi156 ceph-mon[32631]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 63 MiB/s wr, 1.10k op/s 2024-04-25T01:37:41.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:41 smithi016 ceph-mon[29153]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 56 MiB/s wr, 982 op/s 2024-04-25T01:37:41.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:41 smithi038 ceph-mon[32841]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 56 MiB/s wr, 982 op/s 2024-04-25T01:37:41.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:41 smithi156 ceph-mon[32631]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 56 MiB/s wr, 982 op/s 2024-04-25T01:37:43.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:43 smithi038 ceph-mon[32841]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 56 MiB/s wr, 983 op/s 2024-04-25T01:37:43.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:43 smithi156 ceph-mon[32631]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 56 MiB/s wr, 983 op/s 2024-04-25T01:37:44.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:43 smithi016 ceph-mon[29153]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 56 MiB/s wr, 983 op/s 2024-04-25T01:37:45.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:45 smithi038 ceph-mon[32841]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 61 MiB/s wr, 1.06k op/s 2024-04-25T01:37:45.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:45 smithi156 ceph-mon[32631]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 61 MiB/s wr, 1.06k op/s 2024-04-25T01:37:46.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:45 smithi016 ceph-mon[29153]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 61 MiB/s wr, 1.06k op/s 2024-04-25T01:37:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.775+0000 7f10195f9640 1 -- 172.21.15.16:0/298738102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410b0b0 msgr2=0x7f101410d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.775+0000 7f10195f9640 1 --2- 172.21.15.16:0/298738102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410b0b0 0x7f101410d520 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f100c01acb0 tx=0x7f100c03b900 comp rx=0 tx=0).stop 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.778+0000 7f10195f9640 1 -- 172.21.15.16:0/298738102 shutdown_connections 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.778+0000 7f10195f9640 1 --2- 172.21.15.16:0/298738102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101410da60 0x7f101410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.778+0000 7f10195f9640 1 --2- 172.21.15.16:0/298738102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410b0b0 0x7f101410d520 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.778+0000 7f10195f9640 1 --2- 172.21.15.16:0/298738102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1014108740 0x7f101410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.778+0000 7f10195f9640 1 -- 172.21.15.16:0/298738102 >> 172.21.15.16:0/298738102 conn(0x7f101406e7b0 msgr2=0x7f1014070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.779+0000 7f10195f9640 1 -- 172.21.15.16:0/298738102 shutdown_connections 2024-04-25T01:37:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.779+0000 7f10195f9640 1 -- 172.21.15.16:0/298738102 wait complete. 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.780+0000 7f10195f9640 1 Processor -- start 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.780+0000 7f10195f9640 1 -- start start 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.780+0000 7f10195f9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 0x7f101419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.780+0000 7f10195f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f101410b0b0 0x7f101419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.780+0000 7f10195f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410da60 0x7f10141a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.780+0000 7f10195f9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1014111930 con 0x7f101410b0b0 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f10195f9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10141117b0 con 0x7f1014108740 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f10195f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1014111ab0 con 0x7f101410da60 2024-04-25T01:37:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 0x7f101419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f10127fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f101410b0b0 0x7f101419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 0x7f101419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:52738/0 (socket says 172.21.15.16:52738) 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 -- 172.21.15.16:0/4275153568 learned_addr learned my addr 172.21.15.16:0/4275153568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f10137fe640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410da60 0x7f10141a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 -- 172.21.15.16:0/4275153568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410da60 msgr2=0x7f10141a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410da60 0x7f10141a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 -- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f101410b0b0 msgr2=0x7f101419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f101410b0b0 0x7f101419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 -- 172.21.15.16:0/4275153568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10141a5b80 con 0x7f1014108740 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.781+0000 7f1012ffd640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 0x7f101419df60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00b4a0 tx=0x7f0ffc00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.782+0000 7f100bfff640 1 -- 172.21.15.16:0/4275153568 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc014070 con 0x7f1014108740 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.782+0000 7f100bfff640 1 -- 172.21.15.16:0/4275153568 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ffc010ac0 con 0x7f1014108740 2024-04-25T01:37:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.782+0000 7f100bfff640 1 -- 172.21.15.16:0/4275153568 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc017c30 con 0x7f1014108740 2024-04-25T01:37:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.782+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10141a5e70 con 0x7f1014108740 2024-04-25T01:37:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.782+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10141a6300 con 0x7f1014108740 2024-04-25T01:37:46.763 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.785+0000 7f100bfff640 1 -- 172.21.15.16:0/4275153568 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0ffc017070 con 0x7f1014108740 2024-04-25T01:37:46.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 Processor -- start 2024-04-25T01:37:46.764 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 -- start start 2024-04-25T01:37:46.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f1452270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1452840 0x55c9f1457340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1459e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c9f13b5cb0 con 0x55c9f14579b0 2024-04-25T01:37:46.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c9f135e5b0 con 0x55c9f1452840 2024-04-25T01:37:46.765 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.786+0000 7f101c3e2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c9f13b1c70 con 0x55c9f1451e50 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f1452270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f1452270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:59930/0 (socket says 172.21.15.16:59930) 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 -- 172.21.15.16:0/3101351892 learned_addr learned my addr 172.21.15.16:0/3101351892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f1012ffd640 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1452840 0x55c9f1457340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10127fc640 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1459e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 -- 172.21.15.16:0/3101351892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1452840 msgr2=0x55c9f1457340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1452840 0x55c9f1457340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 -- 172.21.15.16:0/3101351892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 msgr2=0x55c9f1459e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1459e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 -- 172.21.15.16:0/3101351892 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9f10ca210 con 0x55c9f1451e50 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f100bfff640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10000a2d00 0x7f10000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.766 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.787+0000 7f10137fe640 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f1452270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1004009ca0 tx=0x7f1004011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c7ea606ab920251 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.788+0000 7f10097fa640 1 -- 172.21.15.16:0/3101351892 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100401e070 con 0x55c9f1451e50 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.788+0000 7f10097fa640 1 -- 172.21.15.16:0/3101351892 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1004009e50 con 0x55c9f1451e50 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.788+0000 7f1012ffd640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10000a2d00 0x7f10000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.788+0000 7f10097fa640 1 -- 172.21.15.16:0/3101351892 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100400ea50 con 0x55c9f1451e50 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.788+0000 7f100bfff640 1 -- 172.21.15.16:0/4275153568 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0ffc0ce620 con 0x7f1014108740 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 -- 172.21.15.16:0/3101351892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 msgr2=0x55c9f1452270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f1452270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1004009ca0 tx=0x7f1004011810 comp rx=0 tx=0).stop 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f1012ffd640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10000a2d00 0x7f10000a51c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0cc470 tx=0x7f0ffc00bf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 -- 172.21.15.16:0/3101351892 shutdown_connections 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1459e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.768 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1452840 0x55c9f1457340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 --2- 172.21.15.16:0/3101351892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f1452270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 -- 172.21.15.16:0/3101351892 >> 172.21.15.16:0/3101351892 conn(0x55c9f144d4c0 msgr2=0x55c9f144d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:46.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 -- 172.21.15.16:0/3101351892 shutdown_connections 2024-04-25T01:37:46.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.789+0000 7f101c3e2740 1 -- 172.21.15.16:0/3101351892 wait complete. 2024-04-25T01:37:46.769 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 -- start start 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 Processor -- start 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f145c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f1452840 0x55c9f145c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 0x55c9f14638e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c9f145aa00 con 0x55c9f1452840 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c9f1371b50 con 0x55c9f14579b0 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c9f145ab80 con 0x55c9f1451e50 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f101c3e2740 10 client.0 _ll_register_callbacks cb 0x55c9f13acca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f10137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 0x55c9f14638e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f10127fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f1452840 0x55c9f145c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f10137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 0x55c9f14638e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:52758/0 (socket says 172.21.15.16:52758) 2024-04-25T01:37:46.771 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.790+0000 7f10137fe640 1 -- 172.21.15.16:0/2851901087 learned_addr learned my addr 172.21.15.16:0/2851901087 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f145c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 -- 172.21.15.16:0/2851901087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 msgr2=0x55c9f145c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f145c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 -- 172.21.15.16:0/2851901087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f1452840 msgr2=0x55c9f145c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f1452840 0x55c9f145c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 -- 172.21.15.16:0/2851901087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9f1196620 con 0x55c9f14579b0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f10137fe640 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 0x55c9f14638e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f100400ea30 tx=0x7f100400e3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cda20f08de8d5705 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f0ffa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.791+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/2851901087 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100402d060 con 0x55c9f14579b0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.792+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1004009e50 con 0x55c9f14579b0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.792+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/2851901087 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1004004030 con 0x55c9f14579b0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.792+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/2851901087 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f100401e070 con 0x55c9f14579b0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.792+0000 7f101c3e2740 10 client.25709 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.792+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c9f1466c50 con 0x55c9f14579b0 2024-04-25T01:37:46.772 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.792+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/2851901087 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1004009e50 con 0x55c9f14579b0 2024-04-25T01:37:46.773 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/2851901087 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f10040035f0 con 0x55c9f14579b0 2024-04-25T01:37:46.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f101c3e2740 10 client.25709 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:46.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f101c3e2740 10 client.25709 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:46.774 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f101c3e2740 20 client.25709 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f101c3e2740 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14686a0 0x55c9f146aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f101c3e2740 4 client.25709 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.793+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9f13c31f0 con 0x55c9f14686a0 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.794+0000 7f10127fc640 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14686a0 0x55c9f146aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.794+0000 7f10127fc640 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14686a0 0x55c9f146aac0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac231a4238981ad6 server_cookie=734a8094f3572893 in_seq=0 out_seq=0 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.795+0000 7f0ffa7fc640 10 client.25709 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.795+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/2851901087 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9f13c31f0 con 0x55c9f14686a0 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.795+0000 7f0ffa7fc640 10 client.25709 handle_command_reply: tid=0 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 client.25709 shutdown 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 msgr2=0x55c9f14638e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 0x55c9f14638e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f100400ea30 tx=0x7f100400e3b0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 shutdown_connections 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14686a0 0x55c9f146aac0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f14579b0 0x55c9f14638e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.775 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f1452840 0x55c9f145c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f0ffa7fc640 0 client.25709 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:46.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 --2- 172.21.15.16:0/2851901087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1451e50 0x55c9f145c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 >> 172.21.15.16:0/2851901087 conn(0x55c9f144d4c0 msgr2=0x55c9f144c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:46.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 shutdown_connections 2024-04-25T01:37:46.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.796+0000 7f101c3e2740 1 -- 172.21.15.16:0/2851901087 wait complete. 2024-04-25T01:37:46.776 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.797+0000 7f101c3e2740 20 client.25709 trim_cache size 0 max 16384 2024-04-25T01:37:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.825+0000 7f101c3e2740 1 Processor -- start 2024-04-25T01:37:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.825+0000 7f101c3e2740 1 -- start start 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f144d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f14638a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1450240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f10137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f144d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f101c3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10141b1be0 con 0x55c9f14579b0 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f10137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f144d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:52760/0 (socket says 172.21.15.16:52760) 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f10137fe640 1 -- 172.21.15.16:0/284597368 learned_addr learned my addr 172.21.15.16:0/284597368 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.826+0000 7f101c3e2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10141af4b0 con 0x55c9f1451e50 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f101c3e2740 1 -- 172.21.15.16:0/284597368 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f10141ad300 con 0x55c9f1452840 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10137fe640 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1450240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10127fc640 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f14638a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.806 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10137fe640 1 -- 172.21.15.16:0/284597368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 msgr2=0x55c9f14638a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10137fe640 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f14638a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10137fe640 1 -- 172.21.15.16:0/284597368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 msgr2=0x55c9f1450240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10137fe640 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1450240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.827+0000 7f10137fe640 1 -- 172.21.15.16:0/284597368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9f14681a0 con 0x55c9f1451e50 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.828+0000 7f10137fe640 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f144d8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1004009ab0 tx=0x7f1004011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36c8254bc0f16776 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.828+0000 7f1008ff9640 1 -- 172.21.15.16:0/284597368 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100402d040 con 0x55c9f1451e50 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.828+0000 7f1008ff9640 1 -- 172.21.15.16:0/284597368 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f100401f040 con 0x55c9f1451e50 2024-04-25T01:37:46.807 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.828+0000 7f1008ff9640 1 -- 172.21.15.16:0/284597368 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10040035f0 con 0x55c9f1451e50 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 -- 172.21.15.16:0/284597368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 msgr2=0x55c9f144d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f144d8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1004009ab0 tx=0x7f1004011810 comp rx=0 tx=0).stop 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 -- 172.21.15.16:0/284597368 shutdown_connections 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f1450240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f14638a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 --2- 172.21.15.16:0/284597368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f144d8a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.829+0000 7f101c3e2740 1 -- 172.21.15.16:0/284597368 >> 172.21.15.16:0/284597368 conn(0x55c9f144d4c0 msgr2=0x55c9f145af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.830+0000 7f101c3e2740 1 -- 172.21.15.16:0/284597368 shutdown_connections 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.830+0000 7f101c3e2740 1 -- 172.21.15.16:0/284597368 wait complete. 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.830+0000 7f101c3e2740 1 -- start start 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.830+0000 7f101c3e2740 1 Processor -- start 2024-04-25T01:37:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.831+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f14606f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.831+0000 7f101c3e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f1460c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.831+0000 7f10137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f14606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.831+0000 7f10137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f14606f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:52766/0 (socket says 172.21.15.16:52766) 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.831+0000 7f10137fe640 1 -- 172.21.15.16:0/928330979 learned_addr learned my addr 172.21.15.16:0/928330979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.831+0000 7f10127fc640 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f1460c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f14ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f10141ad970 con 0x55c9f14579b0 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10141aede0 con 0x55c9f1451e50 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f10141ae040 con 0x55c9f1452840 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f101c3e2740 10 client.0 _ll_register_callbacks cb 0x7f100401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f10127fc640 1 -- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 msgr2=0x55c9f14606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f10127fc640 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f14606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f10127fc640 1 -- 172.21.15.16:0/928330979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 msgr2=0x55c9f14ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f10127fc640 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f14ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.832+0000 7f10127fc640 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9f14cd170 con 0x55c9f1452840 2024-04-25T01:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.833+0000 7f10127fc640 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f1460c30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f100c005e30 tx=0x7f100c003ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7678fcda285ea86e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.833+0000 7f0ffa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T01:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.833+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/928330979 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c05d020 con 0x55c9f1452840 2024-04-25T01:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.833+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/928330979 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f100c040940 con 0x55c9f1452840 2024-04-25T01:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.834+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f100401f040 con 0x55c9f1452840 2024-04-25T01:37:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.834+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/928330979 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f100c068050 con 0x55c9f1452840 2024-04-25T01:37:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.835+0000 7f101c3e2740 10 client.25681 fetch_fsmap learned FSMap version 22 2024-04-25T01:37:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.835+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c9f14ce380 con 0x55c9f1452840 2024-04-25T01:37:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.835+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/928330979 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f100c004040 con 0x55c9f1452840 2024-04-25T01:37:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/928330979 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f100c040b40 con 0x55c9f1452840 2024-04-25T01:37:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f101c3e2740 10 client.25681 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:37:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f101c3e2740 10 client.25681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f101c3e2740 20 client.25681 populate_metadata read hostname 'smithi016' 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14ced30 0x55c9f14d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f101c3e2740 4 client.25681 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.837+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f100401f040 con 0x55c9f14ced30 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.838+0000 7f10137fe640 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14ced30 0x55c9f14d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.838+0000 7f10137fe640 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14ced30 0x55c9f14d1150 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53ed6d5241942e09 server_cookie=dbf69b0a46a2a2a7 in_seq=0 out_seq=0 2024-04-25T01:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.838+0000 7f0ffa7fc640 10 client.25681 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.853+0000 7f0ffa7fc640 1 -- 172.21.15.16:0/928330979 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f100401f040 con 0x55c9f14ced30 2024-04-25T01:37:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.853+0000 7f0ffa7fc640 10 client.25681 handle_command_reply: tid=0 2024-04-25T01:37:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.853+0000 7f101c3e2740 1 client.25681 shutdown 2024-04-25T01:37:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.855+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 msgr2=0x55c9f1460c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.855+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f1460c30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f100c005e30 tx=0x7f100c003ad0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.855+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 shutdown_connections 2024-04-25T01:37:46.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.855+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c9f14579b0 0x55c9f14ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.855+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9f1452840 0x55c9f1460c30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.855+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c9f14ced30 0x55c9f14d1150 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.856+0000 7f101c3e2740 1 --2- 172.21.15.16:0/928330979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9f1451e50 0x55c9f14606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.856+0000 7f0ffa7fc640 0 client.25681 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:37:46.834 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.856+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 >> 172.21.15.16:0/928330979 conn(0x55c9f144d4c0 msgr2=0x55c9f144d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:46.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.856+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 shutdown_connections 2024-04-25T01:37:46.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.856+0000 7f101c3e2740 1 -- 172.21.15.16:0/928330979 wait complete. 2024-04-25T01:37:46.835 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.856+0000 7f101c3e2740 20 client.25681 trim_cache size 0 max 16384 2024-04-25T01:37:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:37:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:37:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:37:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.862+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10000a2d00 msgr2=0x7f10000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.862+0000 7f10195f9640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10000a2d00 0x7f10000a51c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0cc470 tx=0x7f0ffc00bf90 comp rx=0 tx=0).stop 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.862+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 msgr2=0x7f101419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.862+0000 7f10195f9640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 0x7f101419df60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00b4a0 tx=0x7f0ffc00b960 comp rx=0 tx=0).stop 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.862+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 shutdown_connections 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.863+0000 7f10195f9640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f10000a2d00 0x7f10000a51c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.863+0000 7f10195f9640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f101410da60 0x7f10141a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.863+0000 7f10195f9640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f101410b0b0 0x7f101419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.863+0000 7f10195f9640 1 --2- 172.21.15.16:0/4275153568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1014108740 0x7f101419df60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.863+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 >> 172.21.15.16:0/4275153568 conn(0x7f101406e7b0 msgr2=0x7f101410b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.863+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 shutdown_connections 2024-04-25T01:37:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:37:46.864+0000 7f10195f9640 1 -- 172.21.15.16:0/4275153568 wait complete. 2024-04-25T01:37:47.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:47 smithi038 ceph-mon[32841]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 51 MiB/s wr, 880 op/s 2024-04-25T01:37:47.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:47 smithi156 ceph-mon[32631]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 51 MiB/s wr, 880 op/s 2024-04-25T01:37:48.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:47 smithi016 ceph-mon[29153]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 51 MiB/s wr, 880 op/s 2024-04-25T01:37:49.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:49 smithi038 ceph-mon[32841]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 62 MiB/s wr, 1.08k op/s 2024-04-25T01:37:49.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:49 smithi156 ceph-mon[32631]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 62 MiB/s wr, 1.08k op/s 2024-04-25T01:37:50.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:49 smithi016 ceph-mon[29153]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 62 MiB/s wr, 1.08k op/s 2024-04-25T01:37:51.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:51 smithi038 ceph-mon[32841]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 56 MiB/s wr, 968 op/s 2024-04-25T01:37:51.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:51 smithi156 ceph-mon[32631]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 56 MiB/s wr, 968 op/s 2024-04-25T01:37:52.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:51 smithi016 ceph-mon[29153]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 56 MiB/s wr, 968 op/s 2024-04-25T01:37:53.964 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:53 smithi038 ceph-mon[32841]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 57 MiB/s wr, 987 op/s 2024-04-25T01:37:53.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:53 smithi156 ceph-mon[32631]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 57 MiB/s wr, 987 op/s 2024-04-25T01:37:54.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:53 smithi016 ceph-mon[29153]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 57 MiB/s wr, 987 op/s 2024-04-25T01:37:54.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:54 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:54.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:54 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:55.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:54 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:55.965 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:55 smithi038 ceph-mon[32841]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 62 MiB/s wr, 1.08k op/s 2024-04-25T01:37:55.999 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:55 smithi156 ceph-mon[32631]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 62 MiB/s wr, 1.08k op/s 2024-04-25T01:37:56.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:55 smithi016 ceph-mon[29153]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 62 MiB/s wr, 1.08k op/s 2024-04-25T01:37:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:57 smithi038 ceph-mon[32841]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 51 MiB/s wr, 888 op/s 2024-04-25T01:37:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.715 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:57 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:57 smithi156 ceph-mon[32631]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 51 MiB/s wr, 888 op/s 2024-04-25T01:37:57.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.749 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.750 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:57 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:57 smithi016 ceph-mon[29153]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 51 MiB/s wr, 888 op/s 2024-04-25T01:37:57.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:57 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:58 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:58 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:58 smithi016 ceph-mon[29153]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 62 MiB/s wr, 1.07k op/s 2024-04-25T01:37:59.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:58 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:58 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:58 smithi038 ceph-mon[32841]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 62 MiB/s wr, 1.07k op/s 2024-04-25T01:37:59.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:58 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:58 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:37:59.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:58 smithi156 ceph-mon[32631]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 62 MiB/s wr, 1.07k op/s 2024-04-25T01:38:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.048+0000 7f3cff7fe640 1 -- 172.21.15.16:0/3218714415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf40308c0 con 0x7f3d00104bb0 2024-04-25T01:38:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.048+0000 7f3d06f30640 1 -- 172.21.15.16:0/3218714415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 msgr2=0x7f3d00105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.048+0000 7f3d06f30640 1 --2- 172.21.15.16:0/3218714415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 0x7f3d00105030 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009860 tx=0x7f3cf402a380 comp rx=0 tx=0).stop 2024-04-25T01:38:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.049+0000 7f3d06f30640 1 -- 172.21.15.16:0/3218714415 shutdown_connections 2024-04-25T01:38:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.049+0000 7f3d06f30640 1 --2- 172.21.15.16:0/3218714415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d00105570 0x7f3d0010be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.049+0000 7f3d06f30640 1 --2- 172.21.15.16:0/3218714415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 0x7f3d00105030 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.049+0000 7f3d06f30640 1 --2- 172.21.15.16:0/3218714415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d001039c0 0x7f3d00103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.049+0000 7f3d06f30640 1 -- 172.21.15.16:0/3218714415 >> 172.21.15.16:0/3218714415 conn(0x7f3d000fef30 msgr2=0x7f3d00101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.055+0000 7f3d06f30640 1 -- 172.21.15.16:0/3218714415 shutdown_connections 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.055+0000 7f3d06f30640 1 -- 172.21.15.16:0/3218714415 wait complete. 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.056+0000 7f3d06f30640 1 Processor -- start 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.056+0000 7f3d06f30640 1 -- start start 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.056+0000 7f3d06f30640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d001039c0 0x7f3d0006e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.056+0000 7f3d06f30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 0x7f3d0006e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.056+0000 7f3d06f30640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d00105570 0x7f3d0010fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d06f30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d00077560 con 0x7f3d00104bb0 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d06f30640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3d000773e0 con 0x7f3d00105570 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d06f30640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3d000776e0 con 0x7f3d001039c0 2024-04-25T01:38:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d001039c0 0x7f3d0006e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d054cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d00105570 0x7f3d0010fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d001039c0 0x7f3d0006e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38832/0 (socket says 172.21.15.16:38832) 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d04ccc640 1 -- 172.21.15.16:0/2814568598 learned_addr learned my addr 172.21.15.16:0/2814568598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d054cd640 1 -- 172.21.15.16:0/2814568598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d001039c0 msgr2=0x7f3d0006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d054cd640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d001039c0 0x7f3d0006e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d054cd640 1 -- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 msgr2=0x7f3d0006e640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d054cd640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 0x7f3d0006e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.057+0000 7f3d054cd640 1 -- 172.21.15.16:0/2814568598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d001100d0 con 0x7f3d00105570 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.058+0000 7f3d054cd640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d00105570 0x7f3d0010fb90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800ef00 tx=0x7f3cf800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.058+0000 7f3cfdffb640 1 -- 172.21.15.16:0/2814568598 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8004b20 con 0x7f3d00105570 2024-04-25T01:38:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.058+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d001103c0 con 0x7f3d00105570 2024-04-25T01:38:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.058+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d00110910 con 0x7f3d00105570 2024-04-25T01:38:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.058+0000 7f3cfdffb640 1 -- 172.21.15.16:0/2814568598 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cf8003e20 con 0x7f3d00105570 2024-04-25T01:38:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.058+0000 7f3cfdffb640 1 -- 172.21.15.16:0/2814568598 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf800b570 con 0x7f3d00105570 2024-04-25T01:38:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.062+0000 7f3d09c3f740 1 Processor -- start 2024-04-25T01:38:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.062+0000 7f3cfdffb640 1 -- 172.21.15.16:0/2814568598 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3cf801d020 con 0x7f3d00105570 2024-04-25T01:38:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.062+0000 7f3d09c3f740 1 -- start start 2024-04-25T01:38:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7a6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7ab340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c35c709cb0 con 0x55c35c7ab9b0 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c35c6b25b0 con 0x55c35c7a6840 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c35c705c70 con 0x55c35c7a5e50 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7a6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:38844/0 (socket says 172.21.15.16:38844) 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3cfffff640 1 -- 172.21.15.16:0/1326969034 learned_addr learned my addr 172.21.15.16:0/1326969034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3cfffff640 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1326969034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 msgr2=0x55c35c7a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1326969034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 msgr2=0x55c35c7ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3cfdffb640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3cf00a2d00 0x7f3cf00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1326969034 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c35c41e210 con 0x55c35c7a6840 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.063+0000 7f3cfffff640 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.064+0000 7f3cfdffb640 1 -- 172.21.15.16:0/2814568598 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3cf80cee60 con 0x7f3d00105570 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.064+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7ab340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009a10 tx=0x7f3cec009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=434f99bb30ae6cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.064+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3cf00a2d00 0x7f3cf00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.064+0000 7f3ceaffd640 1 -- 172.21.15.16:0/1326969034 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec017070 con 0x55c35c7a6840 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.064+0000 7f3ceaffd640 1 -- 172.21.15.16:0/1326969034 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cec013470 con 0x55c35c7a6840 2024-04-25T01:38:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.064+0000 7f3ceaffd640 1 -- 172.21.15.16:0/1326969034 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec012630 con 0x55c35c7a6840 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3cf00a2d00 0x7f3cf00a51c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009780 tx=0x7f3cf4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1326969034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 msgr2=0x55c35c7ab340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7ab340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009a10 tx=0x7f3cec009ed0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1326969034 shutdown_connections 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7ab340 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1326969034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1326969034 >> 172.21.15.16:0/1326969034 conn(0x55c35c7a14c0 msgr2=0x55c35c7a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1326969034 shutdown_connections 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1326969034 wait complete. 2024-04-25T01:38:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.065+0000 7f3d09c3f740 1 -- start start 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 Processor -- start 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c35c7ae960 con 0x55c35c7ab9b0 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c35c6c5b50 con 0x55c35c7a6840 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c35c7aeae0 con 0x55c35c7a5e50 2024-04-25T01:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:49848/0 (socket says 172.21.15.16:49848) 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3cfffff640 1 -- 172.21.15.16:0/2028878282 learned_addr learned my addr 172.21.15.16:0/2028878282 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.066+0000 7f3d09c3f740 10 client.0 _ll_register_callbacks cb 0x55c35c700ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cfffff640 1 -- 172.21.15.16:0/2028878282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 msgr2=0x55c35c7b02d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7b02d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cfffff640 1 -- 172.21.15.16:0/2028878282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 msgr2=0x55c35c7b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3d054cd640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cfffff640 1 -- 172.21.15.16:0/2028878282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c35c4ea620 con 0x55c35c7a6840 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3d054cd640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b0810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009860 tx=0x7f3cf402a380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1474dd3fb93b23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.067+0000 7f3cdffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.068+0000 7f3cdffff640 1 -- 172.21.15.16:0/2028878282 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf404d070 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.068+0000 7f3cdffff640 1 -- 172.21.15.16:0/2028878282 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3cf4004ad0 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.068+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cec013470 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.068+0000 7f3cdffff640 1 -- 172.21.15.16:0/2028878282 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3cf4057050 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.068+0000 7f3d09c3f740 10 client.25724 fetch_fsmap learned FSMap version 22 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.068+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c35c7bac60 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3cdffff640 1 -- 172.21.15.16:0/2028878282 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cf4038040 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3cdffff640 1 -- 172.21.15.16:0/2028878282 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3cf4049400 con 0x55c35c7a6840 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3d09c3f740 10 client.25724 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3d09c3f740 10 client.25724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3d09c3f740 20 client.25724 populate_metadata read hostname 'smithi016' 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c7be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3d09c3f740 4 client.25724 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.069+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3cec013470 con 0x55c35c7bc3b0 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.070+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c7be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.071+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c7be7e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bb9735c89a5a6b server_cookie=b280f205152148bc in_seq=0 out_seq=0 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.071+0000 7f3cdffff640 10 client.25724 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.072+0000 7f3cdffff640 1 -- 172.21.15.16:0/2028878282 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f3cec013470 con 0x55c35c7bc3b0 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.072+0000 7f3cdffff640 10 client.25724 handle_command_reply: tid=0 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.072+0000 7f3d09c3f740 1 client.25724 shutdown 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.072+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 msgr2=0x55c35c7b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.072+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b0810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009860 tx=0x7f3cf402a380 comp rx=0 tx=0).stop 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 shutdown_connections 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c7be7e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b0810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3cdffff640 0 client.25724 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2028878282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7a5e50 0x55c35c7b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 >> 172.21.15.16:0/2028878282 conn(0x55c35c7a14c0 msgr2=0x55c35c7a05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 shutdown_connections 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2028878282 wait complete. 2024-04-25T01:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.073+0000 7f3d09c3f740 20 client.25724 trim_cache size 0 max 16384 2024-04-25T01:38:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.103+0000 7f3d09c3f740 1 Processor -- start 2024-04-25T01:38:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.103+0000 7f3d09c3f740 1 -- start start 2024-04-25T01:38:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7a4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4010eb0 con 0x55c35c7a5e50 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce400e780 con 0x55c35c7a6840 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce400c5d0 con 0x55c35c7ab9b0 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35970/0 (socket says 172.21.15.16:35970) 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1403288104 learned_addr learned my addr 172.21.15.16:0/1403288104 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3cfffff640 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d054cd640 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1403288104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 msgr2=0x55c35c7a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1403288104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 msgr2=0x55c35c7b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.104+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.105+0000 7f3d04ccc640 1 -- 172.21.15.16:0/1403288104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c35c7a2230 con 0x55c35c7a5e50 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.105+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7afcb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009960 tx=0x7f3cec009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152f4cb9d3efb326 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.106+0000 7f3cea7fc640 1 -- 172.21.15.16:0/1403288104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec04b070 con 0x55c35c7a5e50 2024-04-25T01:38:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.106+0000 7f3cea7fc640 1 -- 172.21.15.16:0/1403288104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cec046dd0 con 0x55c35c7a5e50 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.106+0000 7f3cea7fc640 1 -- 172.21.15.16:0/1403288104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec017050 con 0x55c35c7a5e50 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.106+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1403288104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 msgr2=0x55c35c7afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.106+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7afcb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009960 tx=0x7f3cec009190 comp rx=0 tx=0).stop 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.107+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1403288104 shutdown_connections 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.107+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.107+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.107+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/1403288104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7afcb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.107+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1403288104 >> 172.21.15.16:0/1403288104 conn(0x55c35c7a14c0 msgr2=0x55c35c7af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.107+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1403288104 shutdown_connections 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 -- 172.21.15.16:0/1403288104 wait complete. 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 -- start start 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 Processor -- start 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce400cc40 con 0x55c35c7a5e50 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce400e0b0 con 0x55c35c7a6840 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.108+0000 7f3d09c3f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce400d310 con 0x55c35c7ab9b0 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d09c3f740 10 client.0 _ll_register_callbacks cb 0x7f3cf40096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7b4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35984/0 (socket says 172.21.15.16:35984) 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 -- 172.21.15.16:0/2956876230 learned_addr learned my addr 172.21.15.16:0/2956876230 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 -- 172.21.15.16:0/2956876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 msgr2=0x55c35c7b63d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b63d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 -- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 msgr2=0x55c35c7b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.109+0000 7f3d04ccc640 1 -- 172.21.15.16:0/2956876230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c35c7b69c0 con 0x55c35c7a5e50 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.110+0000 7f3d04ccc640 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7b4f40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00efd0 tx=0x7f3cec009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9341cbe53c5d403 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.110+0000 7f3cdffff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.110+0000 7f3cdffff640 1 -- 172.21.15.16:0/2956876230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec054060 con 0x55c35c7a5e50 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.111+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cec046dd0 con 0x55c35c7a5e50 2024-04-25T01:38:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.111+0000 7f3cdffff640 1 -- 172.21.15.16:0/2956876230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3cec005990 con 0x55c35c7a5e50 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.111+0000 7f3cdffff640 1 -- 172.21.15.16:0/2956876230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3cec017020 con 0x55c35c7a5e50 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.111+0000 7f3d09c3f740 10 client.16044 fetch_fsmap learned FSMap version 22 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.111+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c35c844970 con 0x55c35c7a5e50 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3cdffff640 1 -- 172.21.15.16:0/2956876230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cec046dd0 con 0x55c35c7a5e50 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3cdffff640 1 -- 172.21.15.16:0/2956876230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3cec04e030 con 0x55c35c7a5e50 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3d09c3f740 10 client.16044 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3d09c3f740 10 client.16044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3d09c3f740 20 client.16044 populate_metadata read hostname 'smithi016' 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c846e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3d09c3f740 4 client.16044 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.112+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c35c7171f0 con 0x55c35c7bc3b0 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.113+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c846e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.113+0000 7f3cdffff640 10 client.16044 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.113+0000 7f3cfffff640 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c846e90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4da9840bb1936415 server_cookie=69522d6a7f005490 in_seq=0 out_seq=0 2024-04-25T01:38:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.118+0000 7f3cdffff640 1 -- 172.21.15.16:0/2956876230 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c35c7171f0 con 0x55c35c7bc3b0 2024-04-25T01:38:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.118+0000 7f3cdffff640 10 client.16044 handle_command_reply: tid=0 2024-04-25T01:38:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.118+0000 7f3d09c3f740 1 client.16044 shutdown 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 msgr2=0x55c35c7b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7b4f40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00efd0 tx=0x7f3cec009490 comp rx=0 tx=0).stop 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 shutdown_connections 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c35c7ab9b0 0x55c35c7b63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x55c35c7bc3b0 0x55c35c846e90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c35c7a6840 0x55c35c7b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 --2- 172.21.15.16:0/2956876230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c35c7a5e50 0x55c35c7b4f40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3cdffff640 0 client.16044 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.119+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 >> 172.21.15.16:0/2956876230 conn(0x55c35c7a14c0 msgr2=0x55c35c79fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.120+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 shutdown_connections 2024-04-25T01:38:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.120+0000 7f3d09c3f740 1 -- 172.21.15.16:0/2956876230 wait complete. 2024-04-25T01:38:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.120+0000 7f3d09c3f740 20 client.16044 trim_cache size 0 max 16384 2024-04-25T01:38:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:38:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:38:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:38:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.121+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3cf00a2d00 msgr2=0x7f3cf00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.121+0000 7f3d06f30640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3cf00a2d00 0x7f3cf00a51c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009780 tx=0x7f3cf4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.121+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d00105570 msgr2=0x7f3d0010fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.121+0000 7f3d06f30640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d00105570 0x7f3d0010fb90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800ef00 tx=0x7f3cf800c740 comp rx=0 tx=0).stop 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 shutdown_connections 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f3cf00a2d00 0x7f3cf00a51c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d00105570 0x7f3d0010fb90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d00104bb0 0x7f3d0006e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 --2- 172.21.15.16:0/2814568598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d001039c0 0x7f3d0006e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 >> 172.21.15.16:0/2814568598 conn(0x7f3d000fef30 msgr2=0x7f3d000ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 shutdown_connections 2024-04-25T01:38:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:00.122+0000 7f3d06f30640 1 -- 172.21.15.16:0/2814568598 wait complete. 2024-04-25T01:38:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:38:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:38:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:37:59 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:38:00.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:38:00.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:38:00.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:37:59 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:38:00.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:38:00.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:38:00.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:37:59 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:38:01.109 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:00 smithi016 ceph-mon[29153]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 55 MiB/s wr, 945 op/s 2024-04-25T01:38:01.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:00 smithi038 ceph-mon[32841]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 55 MiB/s wr, 945 op/s 2024-04-25T01:38:01.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:00 smithi156 ceph-mon[32631]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 55 MiB/s wr, 945 op/s 2024-04-25T01:38:03.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:02 smithi038 ceph-mon[32841]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 56 MiB/s wr, 959 op/s 2024-04-25T01:38:03.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:02 smithi156 ceph-mon[32631]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 56 MiB/s wr, 959 op/s 2024-04-25T01:38:03.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:02 smithi016 ceph-mon[29153]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 56 MiB/s wr, 959 op/s 2024-04-25T01:38:05.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:04 smithi038 ceph-mon[32841]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 61 MiB/s wr, 1.04k op/s 2024-04-25T01:38:05.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:04 smithi156 ceph-mon[32631]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 61 MiB/s wr, 1.04k op/s 2024-04-25T01:38:05.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:04 smithi016 ceph-mon[29153]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 61 MiB/s wr, 1.04k op/s 2024-04-25T01:38:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:06 smithi016 ceph-mon[29153]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 50 MiB/s wr, 848 op/s 2024-04-25T01:38:07.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:06 smithi038 ceph-mon[32841]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 50 MiB/s wr, 848 op/s 2024-04-25T01:38:07.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:06 smithi156 ceph-mon[32631]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 50 MiB/s wr, 848 op/s 2024-04-25T01:38:09.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:08 smithi038 ceph-mon[32841]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:09.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:08 smithi156 ceph-mon[32631]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:09.321 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:08 smithi016 ceph-mon[29153]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:11.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:10 smithi038 ceph-mon[32841]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 55 MiB/s wr, 922 op/s 2024-04-25T01:38:11.241 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:10 smithi016 ceph-mon[29153]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 55 MiB/s wr, 922 op/s 2024-04-25T01:38:11.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:10 smithi156 ceph-mon[32631]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 55 MiB/s wr, 922 op/s 2024-04-25T01:38:13.215 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:12 smithi038 ceph-mon[32841]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 56 MiB/s wr, 942 op/s 2024-04-25T01:38:13.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:12 smithi156 ceph-mon[32631]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 56 MiB/s wr, 942 op/s 2024-04-25T01:38:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 -- 172.21.15.16:0/1327205489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 msgr2=0x7fa488078580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 --2- 172.21.15.16:0/1327205489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa488078580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa480009ca0 tx=0x7fa48002a8d0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 -- 172.21.15.16:0/1327205489 shutdown_connections 2024-04-25T01:38:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 --2- 172.21.15.16:0/1327205489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa488078580 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 --2- 172.21.15.16:0/1327205489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 0x7fa4880777a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 --2- 172.21.15.16:0/1327205489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa488076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.293+0000 7fa48c857640 1 -- 172.21.15.16:0/1327205489 >> 172.21.15.16:0/1327205489 conn(0x7fa4880fed70 msgr2=0x7fa4881011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.297+0000 7fa48c857640 1 -- 172.21.15.16:0/1327205489 shutdown_connections 2024-04-25T01:38:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.298+0000 7fa48c857640 1 -- 172.21.15.16:0/1327205489 wait complete. 2024-04-25T01:38:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.298+0000 7fa48c857640 1 Processor -- start 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 -- start start 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa48810e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 0x7fa48810e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa4881aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa4881137c0 con 0x7fa488076100 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa488113640 con 0x7fa488077ce0 2024-04-25T01:38:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.299+0000 7fa48c857640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa488113940 con 0x7fa488077340 2024-04-25T01:38:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa48810e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa48810e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42240/0 (socket says 172.21.15.16:42240) 2024-04-25T01:38:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 -- 172.21.15.16:0/1194832245 learned_addr learned my addr 172.21.15.16:0/1194832245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa48759e640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa4881aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa48659c640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 0x7fa48810e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 -- 172.21.15.16:0/1194832245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 msgr2=0x7fa48810e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 0x7fa48810e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 -- 172.21.15.16:0/1194832245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 msgr2=0x7fa4881aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa4881aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa486d9d640 1 -- 172.21.15.16:0/1194832245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48810ed20 con 0x7fa488076100 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa48759e640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa4881aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.300+0000 7fa48659c640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 0x7fa48810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:13.281 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.301+0000 7fa486d9d640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa48810e170 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa47800b4a0 tx=0x7fa47800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:13.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.301+0000 7fa46ffff640 1 -- 172.21.15.16:0/1194832245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478014070 con 0x7fa488076100 2024-04-25T01:38:13.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.302+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa48806b220 con 0x7fa488076100 2024-04-25T01:38:13.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.302+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48806b7f0 con 0x7fa488076100 2024-04-25T01:38:13.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.302+0000 7fa46ffff640 1 -- 172.21.15.16:0/1194832245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa478010ac0 con 0x7fa488076100 2024-04-25T01:38:13.282 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.302+0000 7fa46ffff640 1 -- 172.21.15.16:0/1194832245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478017c30 con 0x7fa488076100 2024-04-25T01:38:13.285 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.306+0000 7fa46ffff640 1 -- 172.21.15.16:0/1194832245 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa478017070 con 0x7fa488076100 2024-04-25T01:38:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.307+0000 7fa46ffff640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4700a2d50 0x7fa4700a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.307+0000 7fa46ffff640 1 -- 172.21.15.16:0/1194832245 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa4780cf2f0 con 0x7fa488076100 2024-04-25T01:38:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.307+0000 7fa48659c640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4700a2d50 0x7fa4700a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 Processor -- start 2024-04-25T01:38:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 -- start start 2024-04-25T01:38:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452e9fc0 0x5628452ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x5628452ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 0x5628452f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56284524dcb0 con 0x5628452e9fc0 2024-04-25T01:38:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5628451f65b0 con 0x5628452ea9b0 2024-04-25T01:38:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.308+0000 7fa48fd64740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562845249c70 con 0x5628452efb20 2024-04-25T01:38:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 0x5628452f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 0x5628452f1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35188/0 (socket says 172.21.15.16:35188) 2024-04-25T01:38:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 -- 172.21.15.16:0/3440537762 learned_addr learned my addr 172.21.15.16:0/3440537762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 -- 172.21.15.16:0/3440537762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 msgr2=0x5628452ef4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:38:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x5628452ef4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 -- 172.21.15.16:0/3440537762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452e9fc0 msgr2=0x5628452ea3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:38:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.309+0000 7fa48759e640 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452e9fc0 0x5628452ea3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.310+0000 7fa48759e640 1 -- 172.21.15.16:0/3440537762 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562844f62210 con 0x5628452efb20 2024-04-25T01:38:13.295 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.310+0000 7fa48759e640 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 0x5628452f1fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa480009ca0 tx=0x7fa48002f480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67d7bd82ea5883a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.310+0000 7fa46d7fa640 1 -- 172.21.15.16:0/3440537762 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480031460 con 0x5628452efb20 2024-04-25T01:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.310+0000 7fa46d7fa640 1 -- 172.21.15.16:0/3440537762 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa480036040 con 0x5628452efb20 2024-04-25T01:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.310+0000 7fa46d7fa640 1 -- 172.21.15.16:0/3440537762 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4800046e0 con 0x5628452efb20 2024-04-25T01:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.311+0000 7fa48fd64740 1 -- 172.21.15.16:0/3440537762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 msgr2=0x5628452f1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.311+0000 7fa48fd64740 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 0x5628452f1fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa480009ca0 tx=0x7fa48002f480 comp rx=0 tx=0).stop 2024-04-25T01:38:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.311+0000 7fa48fd64740 1 -- 172.21.15.16:0/3440537762 shutdown_connections 2024-04-25T01:38:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.311+0000 7fa48fd64740 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452efb20 0x5628452f1fb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.312+0000 7fa48fd64740 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x5628452ef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.312+0000 7fa48fd64740 1 --2- 172.21.15.16:0/3440537762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452e9fc0 0x5628452ea3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.312+0000 7fa48fd64740 1 -- 172.21.15.16:0/3440537762 >> 172.21.15.16:0/3440537762 conn(0x5628452e5630 msgr2=0x5628452e5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.312+0000 7fa48659c640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4700a2d50 0x7fa4700a5210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa48810f680 tx=0x7fa47c005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.317+0000 7fa48fd64740 1 -- 172.21.15.16:0/3440537762 shutdown_connections 2024-04-25T01:38:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.317+0000 7fa48fd64740 1 -- 172.21.15.16:0/3440537762 wait complete. 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.317+0000 7fa48fd64740 1 -- start start 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.317+0000 7fa48fd64740 1 Processor -- start 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x5628452f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 0x5628452fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5628452f2a70 con 0x5628452efb20 2024-04-25T01:38:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562845209b50 con 0x5628452f9a80 2024-04-25T01:38:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5628452f2bf0 con 0x5628452ea9b0 2024-04-25T01:38:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48fd64740 10 client.0 _ll_register_callbacks cb 0x562845244ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 0x5628452fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 0x5628452fbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:47716/0 (socket says 172.21.15.16:47716) 2024-04-25T01:38:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa48759e640 1 -- 172.21.15.16:0/338802612 learned_addr learned my addr 172.21.15.16:0/338802612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.318+0000 7fa486d9d640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.319+0000 7fa48759e640 1 -- 172.21.15.16:0/338802612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 msgr2=0x5628452f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.319+0000 7fa48759e640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.319+0000 7fa48759e640 1 -- 172.21.15.16:0/338802612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 msgr2=0x5628452f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.319+0000 7fa48759e640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x5628452f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.319+0000 7fa48759e640 1 -- 172.21.15.16:0/338802612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56284502e620 con 0x5628452f9a80 2024-04-25T01:38:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.319+0000 7fa486d9d640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.320+0000 7fa48759e640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 0x5628452fbe70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa48002adf0 tx=0x7fa480004670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a1ff5b0681b5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.320+0000 7fa46aff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.320+0000 7fa46aff5640 1 -- 172.21.15.16:0/338802612 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480077060 con 0x5628452f9a80 2024-04-25T01:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.320+0000 7fa46aff5640 1 -- 172.21.15.16:0/338802612 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa480003b80 con 0x5628452f9a80 2024-04-25T01:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.321+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa480036040 con 0x5628452f9a80 2024-04-25T01:38:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.321+0000 7fa46aff5640 1 -- 172.21.15.16:0/338802612 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa480004860 con 0x5628452f9a80 2024-04-25T01:38:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.321+0000 7fa48fd64740 10 client.25739 fetch_fsmap learned FSMap version 22 2024-04-25T01:38:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.321+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5628452f2a70 con 0x5628452f9a80 2024-04-25T01:38:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.322+0000 7fa46aff5640 1 -- 172.21.15.16:0/338802612 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa480003da0 con 0x5628452f9a80 2024-04-25T01:38:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.322+0000 7fa46aff5640 1 -- 172.21.15.16:0/338802612 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa480066070 con 0x5628452f9a80 2024-04-25T01:38:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa48fd64740 10 client.25739 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:38:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa48fd64740 10 client.25739 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa48fd64740 20 client.25739 populate_metadata read hostname 'smithi016' 2024-04-25T01:38:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa48fd64740 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845300470 0x5628453028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa48fd64740 4 client.25739 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa480036040 con 0x562845300470 2024-04-25T01:38:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.323+0000 7fa486d9d640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845300470 0x5628453028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.324+0000 7fa486d9d640 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845300470 0x5628453028b0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c58a008eb8ca8f server_cookie=a5ad81282d4184b2 in_seq=0 out_seq=0 2024-04-25T01:38:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.324+0000 7fa46aff5640 10 client.25739 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa46aff5640 1 -- 172.21.15.16:0/338802612 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa480036040 con 0x562845300470 2024-04-25T01:38:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa46aff5640 10 client.25739 handle_command_reply: tid=0 2024-04-25T01:38:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 client.25739 shutdown 2024-04-25T01:38:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 msgr2=0x5628452fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 0x5628452fbe70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa48002adf0 tx=0x7fa480004670 comp rx=0 tx=0).stop 2024-04-25T01:38:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 shutdown_connections 2024-04-25T01:38:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845300470 0x5628453028b0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452f9a80 0x5628452fbe70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x5628452f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 --2- 172.21.15.16:0/338802612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 >> 172.21.15.16:0/338802612 conn(0x5628452e5630 msgr2=0x5628452e4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa46aff5640 0 client.25739 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.325+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 shutdown_connections 2024-04-25T01:38:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.326+0000 7fa48fd64740 1 -- 172.21.15.16:0/338802612 wait complete. 2024-04-25T01:38:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.326+0000 7fa48fd64740 20 client.25739 trim_cache size 0 max 16384 2024-04-25T01:38:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.356+0000 7fa48fd64740 1 Processor -- start 2024-04-25T01:38:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.358+0000 7fa48fd64740 1 -- start start 2024-04-25T01:38:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.358+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452efb20 0x5628452e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa486d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa486d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452e8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35204/0 (socket says 172.21.15.16:35204) 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa486d9d640 1 -- 172.21.15.16:0/4207845752 learned_addr learned my addr 172.21.15.16:0/4207845752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa48659c640 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452efb20 0x5628452e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452f9a80 0x5628452e44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.359+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa474010e20 con 0x5628452f9a80 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.360+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa47400e6f0 con 0x5628452efb20 2024-04-25T01:38:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.360+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa47400c540 con 0x5628452ea9b0 2024-04-25T01:38:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.363+0000 7fa48759e640 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452f9a80 0x5628452e44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.366+0000 7fa48659c640 1 -- 172.21.15.16:0/4207845752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 msgr2=0x5628452e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.366+0000 7fa48659c640 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.366+0000 7fa48659c640 1 -- 172.21.15.16:0/4207845752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452f9a80 msgr2=0x5628452e44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.366+0000 7fa48659c640 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452f9a80 0x5628452e44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.366+0000 7fa48659c640 1 -- 172.21.15.16:0/4207845752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628452e63a0 con 0x5628452efb20 2024-04-25T01:38:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.367+0000 7fa48659c640 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452efb20 0x5628452e3fa0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x5628452f59f0 tx=0x7fa47c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=157dfadeadcaf929 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.367+0000 7fa46cff9640 1 -- 172.21.15.16:0/4207845752 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c01a050 con 0x5628452efb20 2024-04-25T01:38:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.368+0000 7fa46cff9640 1 -- 172.21.15.16:0/4207845752 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa47c024070 con 0x5628452efb20 2024-04-25T01:38:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.368+0000 7fa46cff9640 1 -- 172.21.15.16:0/4207845752 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c010630 con 0x5628452efb20 2024-04-25T01:38:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.368+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452efb20 msgr2=0x5628452e3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.369+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452efb20 0x5628452e3fa0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x5628452f59f0 tx=0x7fa47c004730 comp rx=0 tx=0).stop 2024-04-25T01:38:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.370+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 shutdown_connections 2024-04-25T01:38:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.370+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452f9a80 0x5628452e44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.371+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452efb20 0x5628452e3fa0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.371+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4207845752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452ea9b0 0x5628452e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.371+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 >> 172.21.15.16:0/4207845752 conn(0x5628452e5630 msgr2=0x5628452e86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.372+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 shutdown_connections 2024-04-25T01:38:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.373+0000 7fa48fd64740 1 -- 172.21.15.16:0/4207845752 wait complete. 2024-04-25T01:38:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.373+0000 7fa48fd64740 1 -- start start 2024-04-25T01:38:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.373+0000 7fa48fd64740 1 Processor -- start 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x562845311450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x562845311990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452f9a80 0x562845312950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa47400cbb0 con 0x5628452efb20 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa47400e020 con 0x5628452ea9b0 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa486d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x562845311450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa486d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x562845311450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:47730/0 (socket says 172.21.15.16:47730) 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa486d9d640 1 -- 172.21.15.16:0/4027183252 learned_addr learned my addr 172.21.15.16:0/4027183252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa47400d280 con 0x5628452f9a80 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48fd64740 10 client.0 _ll_register_callbacks cb 0x7fa480009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48659c640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x562845311990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa48759e640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452f9a80 0x562845312950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa486d9d640 1 -- 172.21.15.16:0/4027183252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452f9a80 msgr2=0x562845312950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.374+0000 7fa486d9d640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452f9a80 0x562845312950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa486d9d640 1 -- 172.21.15.16:0/4027183252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 msgr2=0x562845311990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa486d9d640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x562845311990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa486d9d640 1 -- 172.21.15.16:0/4027183252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562845312f40 con 0x5628452ea9b0 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa486d9d640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x562845311450 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa478009420 tx=0x7fa47800bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f26bc1bfb74f0c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa46aff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa46aff5640 1 -- 172.21.15.16:0/4027183252 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47801e040 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa46aff5640 1 -- 172.21.15.16:0/4027183252 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa4780cce80 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa47c024070 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.375+0000 7fa46aff5640 1 -- 172.21.15.16:0/4027183252 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa4780aa050 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 10 client.25745 fetch_fsmap learned FSMap version 22 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562845366540 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa46aff5640 1 -- 172.21.15.16:0/4027183252 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa47c024070 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa46aff5640 1 -- 172.21.15.16:0/4027183252 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa4780ba2b0 con 0x5628452ea9b0 2024-04-25T01:38:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 10 client.25745 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 20 client.25745 populate_metadata read hostname 'smithi016' 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845366f70 0x562845369390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 4 client.25745 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56284525b1f0 con 0x562845366f70 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.376+0000 7fa48659c640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845366f70 0x562845369390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.377+0000 7fa48659c640 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845366f70 0x562845369390 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e949c96e316e9ab server_cookie=567fecf3f73cb9db in_seq=0 out_seq=0 2024-04-25T01:38:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.377+0000 7fa46aff5640 10 client.25745 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:13.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:12 smithi016 ceph-mon[29153]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 56 MiB/s wr, 942 op/s 2024-04-25T01:38:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.381+0000 7fa46aff5640 1 -- 172.21.15.16:0/4027183252 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56284525b1f0 con 0x562845366f70 2024-04-25T01:38:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.381+0000 7fa46aff5640 10 client.25745 handle_command_reply: tid=0 2024-04-25T01:38:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.381+0000 7fa48fd64740 1 client.25745 shutdown 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 msgr2=0x562845311450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x562845311450 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa478009420 tx=0x7fa47800bf90 comp rx=0 tx=0).stop 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 shutdown_connections 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628452f9a80 0x562845312950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5628452efb20 0x562845311990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562845366f70 0x562845369390 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 --2- 172.21.15.16:0/4027183252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5628452ea9b0 0x562845311450 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 >> 172.21.15.16:0/4027183252 conn(0x5628452e5630 msgr2=0x5628452f38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.382+0000 7fa46aff5640 0 client.25745 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.383+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 shutdown_connections 2024-04-25T01:38:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.383+0000 7fa48fd64740 1 -- 172.21.15.16:0/4027183252 wait complete. 2024-04-25T01:38:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.384+0000 7fa48fd64740 20 client.25745 trim_cache size 0 max 16384 2024-04-25T01:38:13.363 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:38:13.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:38:13.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:38:13.364 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:38:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4700a2d50 msgr2=0x7fa4700a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4700a2d50 0x7fa4700a5210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa48810f680 tx=0x7fa47c005e50 comp rx=0 tx=0).stop 2024-04-25T01:38:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 msgr2=0x7fa48810e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa48810e170 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa47800b4a0 tx=0x7fa47800b960 comp rx=0 tx=0).stop 2024-04-25T01:38:13.365 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 shutdown_connections 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fa4700a2d50 0x7fa4700a5210 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa488077ce0 0x7fa4881aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa488077340 0x7fa48810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 --2- 172.21.15.16:0/1194832245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa488076100 0x7fa48810e170 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 >> 172.21.15.16:0/1194832245 conn(0x7fa4880fed70 msgr2=0x7fa488100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 shutdown_connections 2024-04-25T01:38:13.366 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:13.386+0000 7fa48c857640 1 -- 172.21.15.16:0/1194832245 wait complete. 2024-04-25T01:38:15.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:14 smithi038 ceph-mon[32841]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 60 MiB/s wr, 1.01k op/s 2024-04-25T01:38:15.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:14 smithi156 ceph-mon[32631]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 60 MiB/s wr, 1.01k op/s 2024-04-25T01:38:15.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:14 smithi016 ceph-mon[29153]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 60 MiB/s wr, 1.01k op/s 2024-04-25T01:38:17.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:16 smithi038 ceph-mon[32841]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 49 MiB/s wr, 829 op/s 2024-04-25T01:38:17.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:16 smithi156 ceph-mon[32631]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 49 MiB/s wr, 829 op/s 2024-04-25T01:38:17.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:16 smithi016 ceph-mon[29153]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 49 MiB/s wr, 829 op/s 2024-04-25T01:38:19.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:18 smithi038 ceph-mon[32841]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 61 MiB/s wr, 1.02k op/s 2024-04-25T01:38:19.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:18 smithi156 ceph-mon[32631]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 61 MiB/s wr, 1.02k op/s 2024-04-25T01:38:19.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:18 smithi016 ceph-mon[29153]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 61 MiB/s wr, 1.02k op/s 2024-04-25T01:38:21.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:20 smithi038 ceph-mon[32841]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 55 MiB/s wr, 919 op/s 2024-04-25T01:38:21.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:20 smithi156 ceph-mon[32631]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 55 MiB/s wr, 919 op/s 2024-04-25T01:38:21.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:20 smithi016 ceph-mon[29153]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 55 MiB/s wr, 919 op/s 2024-04-25T01:38:23.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:22 smithi038 ceph-mon[32841]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 56 MiB/s wr, 945 op/s 2024-04-25T01:38:23.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:22 smithi156 ceph-mon[32631]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 56 MiB/s wr, 945 op/s 2024-04-25T01:38:23.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:22 smithi016 ceph-mon[29153]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 56 MiB/s wr, 945 op/s 2024-04-25T01:38:25.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:24 smithi038 ceph-mon[32841]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:25.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:24 smithi156 ceph-mon[32631]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:25.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:24 smithi016 ceph-mon[29153]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:26.529 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 -- 172.21.15.16:0/1352268925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc101890 msgr2=0x7faffc101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 --2- 172.21.15.16:0/1352268925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc101890 0x7faffc101d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fafec009900 tx=0x7fafec02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 -- 172.21.15.16:0/1352268925 shutdown_connections 2024-04-25T01:38:26.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 --2- 172.21.15.16:0/1352268925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102250 0x7faffc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 --2- 172.21.15.16:0/1352268925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc101890 0x7faffc101d10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 --2- 172.21.15.16:0/1352268925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faffc1006a0 0x7faffc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.530 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 -- 172.21.15.16:0/1352268925 >> 172.21.15.16:0/1352268925 conn(0x7faffc0fbc10 msgr2=0x7faffc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:26.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 -- 172.21.15.16:0/1352268925 shutdown_connections 2024-04-25T01:38:26.531 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.549+0000 7fb003010640 1 -- 172.21.15.16:0/1352268925 wait complete. 2024-04-25T01:38:26.533 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.550+0000 7fb003010640 1 Processor -- start 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.551+0000 7fb003010640 1 -- start start 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.551+0000 7fb003010640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 0x7faffc06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.551+0000 7fb003010640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faffc101890 0x7faffc06f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb003010640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102250 0x7faffc0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb000dac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 0x7faffc06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb000dac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 0x7faffc06ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:36332/0 (socket says 172.21.15.16:36332) 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb000dac640 1 -- 172.21.15.16:0/2652838726 learned_addr learned my addr 172.21.15.16:0/2652838726 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:26.534 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faffc06ba20 con 0x7faffc101890 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7faff3fff640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faffc101890 0x7faffc06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb0015ad640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102250 0x7faffc0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faffc06b8a0 con 0x7faffc102250 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.552+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faffc06bba0 con 0x7faffc1006a0 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.553+0000 7fb000dac640 1 -- 172.21.15.16:0/2652838726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102250 msgr2=0x7faffc0724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.553+0000 7fb000dac640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102250 0x7faffc0724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.553+0000 7fb000dac640 1 -- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faffc101890 msgr2=0x7faffc06f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.553+0000 7fb000dac640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faffc101890 0x7faffc06f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.553+0000 7fb000dac640 1 -- 172.21.15.16:0/2652838726 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc072b50 con 0x7faffc1006a0 2024-04-25T01:38:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.554+0000 7fb000dac640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 0x7faffc06ef50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faff800e9c0 tx=0x7faff800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.556+0000 7faff1ffb640 1 -- 172.21.15.16:0/2652838726 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff800cd80 con 0x7faffc1006a0 2024-04-25T01:38:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.556+0000 7faff1ffb640 1 -- 172.21.15.16:0/2652838726 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faff80035f0 con 0x7faffc1006a0 2024-04-25T01:38:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.556+0000 7faff1ffb640 1 -- 172.21.15.16:0/2652838726 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8010690 con 0x7faffc1006a0 2024-04-25T01:38:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.558+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faffc072e40 con 0x7faffc1006a0 2024-04-25T01:38:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.558+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faffc073340 con 0x7faffc1006a0 2024-04-25T01:38:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.561+0000 7fb005df4740 1 Processor -- start 2024-04-25T01:38:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.561+0000 7fb005df4740 1 -- start start 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb005df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a5e50 0x5627532a6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb005df4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532ab340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb005df4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 0x5627532ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb005df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562753209cb0 con 0x5627532a5e50 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb005df4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5627531b25b0 con 0x5627532ab9b0 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb005df4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562753205c70 con 0x5627532a6840 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7faff3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 0x5627532ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7faff3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 0x5627532ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.16:43394/0 (socket says 172.21.15.16:43394) 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7faff3fff640 1 -- 172.21.15.16:0/1079152115 learned_addr learned my addr 172.21.15.16:0/1079152115 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.562+0000 7fb0015ad640 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a5e50 0x5627532a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.563+0000 7faff3fff640 1 -- 172.21.15.16:0/1079152115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 msgr2=0x5627532ab340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:38:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.563+0000 7faff3fff640 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532ab340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.563+0000 7faff3fff640 1 -- 172.21.15.16:0/1079152115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a5e50 msgr2=0x5627532a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.563+0000 7faff3fff640 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a5e50 0x5627532a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.563+0000 7faff3fff640 1 -- 172.21.15.16:0/1079152115 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562752f1e210 con 0x5627532ab9b0 2024-04-25T01:38:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.564+0000 7faff3fff640 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 0x5627532ade40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fafec009900 tx=0x7fafec004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2750a2632068453e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.565+0000 7fafe2ffd640 1 -- 172.21.15.16:0/1079152115 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec035040 con 0x5627532ab9b0 2024-04-25T01:38:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.565+0000 7fafe2ffd640 1 -- 172.21.15.16:0/1079152115 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafec004720 con 0x5627532ab9b0 2024-04-25T01:38:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.566+0000 7fb005df4740 1 -- 172.21.15.16:0/1079152115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 msgr2=0x5627532ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.566+0000 7fb005df4740 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 0x5627532ade40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fafec009900 tx=0x7fafec004040 comp rx=0 tx=0).stop 2024-04-25T01:38:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.566+0000 7faff1ffb640 1 -- 172.21.15.16:0/2652838726 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7faff80108f0 con 0x7faffc1006a0 2024-04-25T01:38:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.567+0000 7faff1ffb640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fafe40a2e70 0x7fafe40a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.567+0000 7faff1ffb640 1 -- 172.21.15.16:0/2652838726 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faff80cf350 con 0x7faffc1006a0 2024-04-25T01:38:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.573+0000 7fb005df4740 1 -- 172.21.15.16:0/1079152115 shutdown_connections 2024-04-25T01:38:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.573+0000 7fb005df4740 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532ab9b0 0x5627532ade40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.573+0000 7fb005df4740 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.573+0000 7fb005df4740 1 --2- 172.21.15.16:0/1079152115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a5e50 0x5627532a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.573+0000 7fb005df4740 1 -- 172.21.15.16:0/1079152115 >> 172.21.15.16:0/1079152115 conn(0x5627532a14c0 msgr2=0x5627532a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:26.553 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.573+0000 7fb005df4740 1 -- 172.21.15.16:0/1079152115 shutdown_connections 2024-04-25T01:38:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.575+0000 7fb005df4740 1 -- 172.21.15.16:0/1079152115 wait complete. 2024-04-25T01:38:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.575+0000 7fb005df4740 1 -- start start 2024-04-25T01:38:26.555 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.575+0000 7fb005df4740 1 Processor -- start 2024-04-25T01:38:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb0015ad640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fafe40a2e70 0x7fafe40a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532ab9b0 0x5627532b7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5627532ae950 con 0x5627532ab9b0 2024-04-25T01:38:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5627531c5b50 con 0x5627532a5e50 2024-04-25T01:38:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5627532aead0 con 0x5627532a6840 2024-04-25T01:38:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb005df4740 10 client.0 _ll_register_callbacks cb 0x562753200ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb0015ad640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fafe40a2e70 0x7fafe40a5330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5627532a70a0 tx=0x7faff4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.576+0000 7fb0015ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532ab9b0 0x5627532b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.577+0000 7faff3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.577+0000 7fb0015ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532ab9b0 0x5627532b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48666/0 (socket says 172.21.15.16:48666) 2024-04-25T01:38:26.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.577+0000 7fb0015ad640 1 -- 172.21.15.16:0/2235135143 learned_addr learned my addr 172.21.15.16:0/2235135143 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:26.559 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.577+0000 7fb000dac640 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.577+0000 7fb000dac640 1 -- 172.21.15.16:0/2235135143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 msgr2=0x5627532b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fb000dac640 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fb000dac640 1 -- 172.21.15.16:0/2235135143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532ab9b0 msgr2=0x5627532b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fb000dac640 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532ab9b0 0x5627532b7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fb000dac640 1 -- 172.21.15.16:0/2235135143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562752fea620 con 0x5627532a5e50 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fb000dac640 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b0320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5627532a72b0 tx=0x7faff809b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b02928107b68c828 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fafdbfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.578+0000 7fafdbfff640 1 -- 172.21.15.16:0/2235135143 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff80b9060 con 0x5627532a5e50 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.579+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafec004720 con 0x5627532a5e50 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.579+0000 7fafdbfff640 1 -- 172.21.15.16:0/2235135143 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faff8010be0 con 0x5627532a5e50 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.579+0000 7fafdbfff640 1 -- 172.21.15.16:0/2235135143 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faff80b4070 con 0x5627532a5e50 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.579+0000 7fb005df4740 10 client.25757 fetch_fsmap learned FSMap version 22 2024-04-25T01:38:26.561 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.579+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5627532bac60 con 0x5627532a5e50 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.579+0000 7fafdbfff640 1 -- 172.21.15.16:0/2235135143 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafec004720 con 0x5627532a5e50 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fafdbfff640 1 -- 172.21.15.16:0/2235135143 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faff80a4310 con 0x5627532a5e50 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fb005df4740 10 client.25757 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fb005df4740 10 client.25757 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fb005df4740 20 client.25757 populate_metadata read hostname 'smithi016' 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fb005df4740 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5627532bc7c0 0x5627532bebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fb005df4740 4 client.25757 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627532171f0 con 0x5627532bc7c0 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.580+0000 7faff3fff640 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5627532bc7c0 0x5627532bebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.562 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.581+0000 7fafdbfff640 10 client.25757 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.581+0000 7faff3fff640 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5627532bc7c0 0x5627532bebd0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=683bc49d36e4df8b server_cookie=eadb35eb0ff899f8 in_seq=0 out_seq=0 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.582+0000 7fafdbfff640 1 -- 172.21.15.16:0/2235135143 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627532171f0 con 0x5627532bc7c0 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.582+0000 7fafdbfff640 10 client.25757 handle_command_reply: tid=0 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 client.25757 shutdown 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 msgr2=0x5627532b0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b0320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5627532a72b0 tx=0x7faff809b7a0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 shutdown_connections 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532ab9b0 0x5627532b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532a6840 0x5627532b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x5627532bc7c0 0x5627532bebd0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 --2- 172.21.15.16:0/2235135143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b0320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 >> 172.21.15.16:0/2235135143 conn(0x5627532a14c0 msgr2=0x5627532a05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fafdbfff640 0 client.25757 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:26.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 shutdown_connections 2024-04-25T01:38:26.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.583+0000 7fb005df4740 1 -- 172.21.15.16:0/2235135143 wait complete. 2024-04-25T01:38:26.565 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.584+0000 7fb005df4740 20 client.25757 trim_cache size 0 max 16384 2024-04-25T01:38:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.611+0000 7fb005df4740 1 Processor -- start 2024-04-25T01:38:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.611+0000 7fb005df4740 1 -- start start 2024-04-25T01:38:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.612+0000 7fb005df4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532aff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.612+0000 7fb005df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.612+0000 7fb005df4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb005df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faffc1ba0e0 con 0x5627532a6840 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb005df4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faffc1b79b0 con 0x5627532a5e50 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb005df4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faffc1b5800 con 0x5627532ab9b0 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb0015ad640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb0015ad640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:36342/0 (socket says 172.21.15.16:36342) 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7faff3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb0015ad640 1 -- 172.21.15.16:0/1259970342 learned_addr learned my addr 172.21.15.16:0/1259970342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7fb000dac640 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7faff3fff640 1 -- 172.21.15.16:0/1259970342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 msgr2=0x5627532b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7faff3fff640 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7faff3fff640 1 -- 172.21.15.16:0/1259970342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 msgr2=0x5627532aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7faff3fff640 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532aff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.613+0000 7faff3fff640 1 -- 172.21.15.16:0/1259970342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627532a1060 con 0x5627532a6840 2024-04-25T01:38:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.614+0000 7faff3fff640 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b77f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fafec005ab0 tx=0x7fafec004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b1178f8242089e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.614+0000 7fafe27fc640 1 -- 172.21.15.16:0/1259970342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec056070 con 0x5627532a6840 2024-04-25T01:38:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.614+0000 7fafe27fc640 1 -- 172.21.15.16:0/1259970342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafec004720 con 0x5627532a6840 2024-04-25T01:38:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.615+0000 7fafe27fc640 1 -- 172.21.15.16:0/1259970342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec035040 con 0x5627532a6840 2024-04-25T01:38:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 -- 172.21.15.16:0/1259970342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 msgr2=0x5627532b77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b77f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fafec005ab0 tx=0x7fafec004610 comp rx=0 tx=0).stop 2024-04-25T01:38:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 -- 172.21.15.16:0/1259970342 shutdown_connections 2024-04-25T01:38:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b77f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 --2- 172.21.15.16:0/1259970342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.616+0000 7fb005df4740 1 -- 172.21.15.16:0/1259970342 >> 172.21.15.16:0/1259970342 conn(0x5627532a14c0 msgr2=0x5627532a18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.617+0000 7fb005df4740 1 -- 172.21.15.16:0/1259970342 shutdown_connections 2024-04-25T01:38:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.617+0000 7fb005df4740 1 -- 172.21.15.16:0/1259970342 wait complete. 2024-04-25T01:38:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.617+0000 7fb005df4740 1 -- start start 2024-04-25T01:38:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.618+0000 7fb005df4740 1 Processor -- start 2024-04-25T01:38:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.618+0000 7fb005df4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.618+0000 7fb005df4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.618+0000 7fb005df4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7fb005df4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faffc1b5e70 con 0x5627532a6840 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7fb005df4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faffc1b72e0 con 0x5627532a5e50 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7fb005df4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faffc1b6540 con 0x5627532ab9b0 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7fb005df4740 10 client.0 _ll_register_callbacks cb 0x7faff80046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7faff3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7faff3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48678/0 (socket says 172.21.15.16:48678) 2024-04-25T01:38:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.619+0000 7faff3fff640 1 -- 172.21.15.16:0/1528762608 learned_addr learned my addr 172.21.15.16:0/1528762608 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.620+0000 7faff3fff640 1 -- 172.21.15.16:0/1528762608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 msgr2=0x5627532b46b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:38:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.620+0000 7faff3fff640 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b46b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.620+0000 7faff3fff640 1 -- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 msgr2=0x5627532b7680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:38:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.620+0000 7faff3fff640 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b7680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.620+0000 7faff3fff640 1 -- 172.21.15.16:0/1528762608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562753320d40 con 0x5627532a6840 2024-04-25T01:38:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.620+0000 7faff3fff640 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b4170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fafec0045e0 tx=0x7fafec004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1adc8aef1ed914f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.621+0000 7fafdbfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-25T01:38:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.622+0000 7fafdbfff640 1 -- 172.21.15.16:0/1528762608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec035050 con 0x5627532a6840 2024-04-25T01:38:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.622+0000 7fafdbfff640 1 -- 172.21.15.16:0/1528762608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fafec005040 con 0x5627532a6840 2024-04-25T01:38:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.623+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafec004720 con 0x5627532a6840 2024-04-25T01:38:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.624+0000 7fafdbfff640 1 -- 172.21.15.16:0/1528762608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fafec056040 con 0x5627532a6840 2024-04-25T01:38:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.624+0000 7fb005df4740 10 client.16071 fetch_fsmap learned FSMap version 22 2024-04-25T01:38:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.624+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562753321f50 con 0x5627532a6840 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fafdbfff640 1 -- 172.21.15.16:0/1528762608 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafec004720 con 0x5627532a6840 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fafdbfff640 1 -- 172.21.15.16:0/1528762608 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fafec059070 con 0x5627532a6840 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb005df4740 10 client.16071 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb005df4740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb005df4740 20 client.16071 populate_metadata read hostname 'smithi016' 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb005df4740 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562753322310 0x562753324700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb005df4740 4 client.16071 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 --> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562753211540 con 0x562753322310 2024-04-25T01:38:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.625+0000 7fb000dac640 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562753322310 0x562753324700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.626+0000 7fafdbfff640 10 client.16071 ms_handle_connect on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.626+0000 7fb000dac640 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562753322310 0x562753324700 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815634cb7ff1b589 server_cookie=c6f39dd82c360ef in_seq=0 out_seq=0 2024-04-25T01:38:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.632+0000 7fafdbfff640 1 -- 172.21.15.16:0/1528762608 <== mds.0 v2:172.21.15.38:6838/2790744598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562753211540 con 0x562753322310 2024-04-25T01:38:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.632+0000 7fafdbfff640 10 client.16071 handle_command_reply: tid=0 2024-04-25T01:38:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.632+0000 7fb005df4740 1 client.16071 shutdown 2024-04-25T01:38:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.633+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 msgr2=0x5627532b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.633+0000 7fb005df4740 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b4170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fafec0045e0 tx=0x7fafec004b00 comp rx=0 tx=0).stop 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 shutdown_connections 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627532ab9b0 0x5627532b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5627532a6840 0x5627532b4170 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:6838/2790744598,v1:172.21.15.38:6839/2790744598] conn(0x562753322310 0x562753324700 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 --2- 172.21.15.16:0/1528762608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627532a5e50 0x5627532b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 >> 172.21.15.16:0/1528762608 conn(0x5627532a14c0 msgr2=0x5627532afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fafdbfff640 0 client.16071 ms_handle_reset on v2:172.21.15.38:6838/2790744598 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 shutdown_connections 2024-04-25T01:38:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.634+0000 7fb005df4740 1 -- 172.21.15.16:0/1528762608 wait complete. 2024-04-25T01:38:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.636+0000 7fb005df4740 20 client.16071 trim_cache size 0 max 16384 2024-04-25T01:38:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:{ 2024-04-25T01:38:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "message": "", 2024-04-25T01:38:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout: "return_code": 0 2024-04-25T01:38:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stdout:} 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.639+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fafe40a2e70 msgr2=0x7fafe40a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fafe40a2e70 0x7fafe40a5330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5627532a70a0 tx=0x7faff4009500 comp rx=0 tx=0).stop 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 msgr2=0x7faffc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 0x7faffc06ef50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faff800e9c0 tx=0x7faff800ee80 comp rx=0 tx=0).stop 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 shutdown_connections 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fafe40a2e70 0x7fafe40a5330 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102250 0x7faffc0724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faffc101890 0x7faffc06f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 --2- 172.21.15.16:0/2652838726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faffc1006a0 0x7faffc06ef50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 >> 172.21.15.16:0/2652838726 conn(0x7faffc0fbc10 msgr2=0x7faffc102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.640+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 shutdown_connections 2024-04-25T01:38:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi016.front.sepia.ceph.com.smithi016.stderr:2024-04-25T01:38:26.641+0000 7fb003010640 1 -- 172.21.15.16:0/2652838726 wait complete. 2024-04-25T01:38:27.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:26 smithi038 ceph-mon[32841]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 51 MiB/s wr, 861 op/s 2024-04-25T01:38:27.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:26 smithi156 ceph-mon[32631]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 51 MiB/s wr, 861 op/s 2024-04-25T01:38:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:26 smithi016 ceph-mon[29153]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 51 MiB/s wr, 861 op/s 2024-04-25T01:38:29.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:28 smithi038 ceph-mon[32841]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 59 MiB/s wr, 997 op/s 2024-04-25T01:38:29.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:28 smithi156 ceph-mon[32631]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 59 MiB/s wr, 997 op/s 2024-04-25T01:38:29.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:28 smithi016 ceph-mon[29153]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 59 MiB/s wr, 997 op/s 2024-04-25T01:38:31.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:30 smithi038 ceph-mon[32841]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 56 MiB/s wr, 934 op/s 2024-04-25T01:38:31.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:30 smithi156 ceph-mon[32631]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 56 MiB/s wr, 934 op/s 2024-04-25T01:38:31.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:30 smithi016 ceph-mon[29153]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 56 MiB/s wr, 934 op/s 2024-04-25T01:38:33.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:32 smithi038 ceph-mon[32841]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 57 MiB/s wr, 950 op/s 2024-04-25T01:38:33.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:32 smithi156 ceph-mon[32631]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 57 MiB/s wr, 950 op/s 2024-04-25T01:38:33.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:32 smithi016 ceph-mon[29153]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 57 MiB/s wr, 950 op/s 2024-04-25T01:38:34.630 INFO:tasks.workunit.client.0.smithi016.stdout:stopping iogen 2024-04-25T01:38:34.631 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'stopping iogen' 2024-04-25T01:38:34.631 INFO:tasks.workunit.client.0.smithi016.stderr:+ ./iogen -k 2024-04-25T01:38:34.633 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2024-04-25T01:38:34.634 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2024-04-25T01:38:35.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:34 smithi038 ceph-mon[32841]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:35.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:34 smithi156 ceph-mon[32631]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:35.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:34 smithi016 ceph-mon[29153]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 61 MiB/s wr, 1.03k op/s 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T01:38:36.748 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T01:38:37.217 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:36 smithi038 ceph-mon[32841]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 51 MiB/s wr, 849 op/s 2024-04-25T01:38:37.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:36 smithi156 ceph-mon[32631]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 51 MiB/s wr, 849 op/s 2024-04-25T01:38:37.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:36 smithi016 ceph-mon[29153]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 51 MiB/s wr, 849 op/s 2024-04-25T01:38:37.643 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-04-25T01:38:37.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T01:38:37.945 DEBUG:teuthology.parallel:result is None 2024-04-25T01:38:37.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T01:38:37.983 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T01:38:37.983 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T01:38:37.995 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:38:38.031 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T01:38:38.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:38:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 -- 172.21.15.16:0/500127518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160101890 msgr2=0x7f6160101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 --2- 172.21.15.16:0/500127518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160101890 0x7f6160101d10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6150009900 tx=0x7f615002a410 comp rx=0 tx=0).stop 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 -- 172.21.15.16:0/500127518 shutdown_connections 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 --2- 172.21.15.16:0/500127518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6160102250 0x7f6160108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 --2- 172.21.15.16:0/500127518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160101890 0x7f6160101d10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 --2- 172.21.15.16:0/500127518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61601006a0 0x7f6160100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 -- 172.21.15.16:0/500127518 >> 172.21.15.16:0/500127518 conn(0x7f61600fbc30 msgr2=0x7f61600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 -- 172.21.15.16:0/500127518 shutdown_connections 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 -- 172.21.15.16:0/500127518 wait complete. 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 Processor -- start 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.199+0000 7f6164909640 1 -- start start 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61601006a0 0x7f616006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6160101890 0x7f616019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 0x7f616019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f616010c1b0 con 0x7f6160102250 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f616010c030 con 0x7f61601006a0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f616010c330 con 0x7f6160101890 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 0x7f616019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 0x7f616019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53110/0 (socket says 172.21.15.16:53110) 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 -- 172.21.15.16:0/3846127253 learned_addr learned my addr 172.21.15.16:0/3846127253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615dd9b640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6160101890 0x7f616019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615e59c640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61601006a0 0x7f616006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 -- 172.21.15.16:0/3846127253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6160101890 msgr2=0x7f616019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6160101890 0x7f616019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 -- 172.21.15.16:0/3846127253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61601006a0 msgr2=0x7f616006a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61601006a0 0x7f616006a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 -- 172.21.15.16:0/3846127253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616019f610 con 0x7f6160102250 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f615ed9d640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 0x7f616019ed50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f615800eaa0 tx=0x7f615800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615800cdb0 con 0x7f6160102250 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616019f900 con 0x7f6160102250 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.200+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6158005020 con 0x7f6160102250 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.201+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580116d0 con 0x7f6160102250 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.201+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616019fe20 con 0x7f6160102250 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.201+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6160107b80 con 0x7f6160102250 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.203+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f61580051d0 con 0x7f6160102250 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.203+0000 7f614f7fe640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61480a2b90 0x7f61480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.204+0000 7f615e59c640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61480a2b90 0x7f61480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.204+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f61580cf5b0 con 0x7f6160102250 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.204+0000 7f615e59c640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61480a2b90 0x7f61480a5050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f61601016f0 tx=0x7f615400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.207+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6158016030 con 0x7f6160102250 2024-04-25T01:38:38.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.391+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f616002d210 con 0x7f6160102250 2024-04-25T01:38:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.393+0000 7f614f7fe640 1 -- 172.21.15.16:0/3846127253 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7f615809be40 con 0x7f6160102250 2024-04-25T01:38:38.374 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:38:38.374 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:38:38.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61480a2b90 msgr2=0x7f61480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61480a2b90 0x7f61480a5050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f61601016f0 tx=0x7f615400b390 comp rx=0 tx=0).stop 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 msgr2=0x7f616019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 0x7f616019ed50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f615800eaa0 tx=0x7f615800ef60 comp rx=0 tx=0).stop 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 shutdown_connections 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f61480a2b90 0x7f61480a5050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6160102250 0x7f616019ed50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6160101890 0x7f616019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 --2- 172.21.15.16:0/3846127253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61601006a0 0x7f616006a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 >> 172.21.15.16:0/3846127253 conn(0x7f61600fbc30 msgr2=0x7f6160104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 shutdown_connections 2024-04-25T01:38:38.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.395+0000 7f6164909640 1 -- 172.21.15.16:0/3846127253 wait complete. 2024-04-25T01:38:38.387 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:38:38.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:38:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 -- 172.21.15.16:0/1927370177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504102220 msgr2=0x7f3504104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 --2- 172.21.15.16:0/1927370177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504102220 0x7f3504104610 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009900 tx=0x7f34f402a410 comp rx=0 tx=0).stop 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 -- 172.21.15.16:0/1927370177 shutdown_connections 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 --2- 172.21.15.16:0/1927370177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504107480 0x7f35041098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 --2- 172.21.15.16:0/1927370177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504104b50 0x7f3504106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 --2- 172.21.15.16:0/1927370177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504102220 0x7f3504104610 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 -- 172.21.15.16:0/1927370177 >> 172.21.15.16:0/1927370177 conn(0x7f35040fbc30 msgr2=0x7f35040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 -- 172.21.15.16:0/1927370177 shutdown_connections 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.513+0000 7f350c134640 1 -- 172.21.15.16:0/1927370177 wait complete. 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 Processor -- start 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 -- start start 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504102220 0x7f35041979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 0x7f3504197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504107480 0x7f350419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f350410d320 con 0x7f3504104b50 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f350410d1a0 con 0x7f3504102220 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350c134640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f350410d4a0 con 0x7f3504107480 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 0x7f3504197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 0x7f3504197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53132/0 (socket says 172.21.15.16:53132) 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 -- 172.21.15.16:0/102115616 learned_addr learned my addr 172.21.15.16:0/102115616 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f3509ed0640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504102220 0x7f35041979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350a6d1640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504107480 0x7f350419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 -- 172.21.15.16:0/102115616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504107480 msgr2=0x7f350419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504107480 0x7f350419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 -- 172.21.15.16:0/102115616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504102220 msgr2=0x7f35041979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504102220 0x7f35041979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 -- 172.21.15.16:0/102115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350419f5e0 con 0x7f3504104b50 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f350a6d1640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504107480 0x7f350419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f3509ed0640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504102220 0x7f35041979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.514+0000 7f35096cf640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 0x7f3504197f00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f350000c970 tx=0x7f350000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.515+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500014070 con 0x7f3504104b50 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.515+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350419f8d0 con 0x7f3504104b50 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.515+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3500004440 con 0x7f3504104b50 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.515+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500003e50 con 0x7f3504104b50 2024-04-25T01:38:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.515+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350419fe20 con 0x7f3504104b50 2024-04-25T01:38:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.516+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35040fd720 con 0x7f3504104b50 2024-04-25T01:38:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.517+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3500017070 con 0x7f3504104b50 2024-04-25T01:38:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.517+0000 7f34faffd640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f34f00a2d30 0x7f34f00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.518+0000 7f3509ed0640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f34f00a2d30 0x7f34f00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.518+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f35000cecf0 con 0x7f3504104b50 2024-04-25T01:38:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.518+0000 7f3509ed0640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f34f00a2d30 0x7f34f00a51f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f34f402a930 tx=0x7f34f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.522+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f350001e030 con 0x7f3504104b50 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.707+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3504062340 con 0x7f3504104b50 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.707+0000 7f34faffd640 1 -- 172.21.15.16:0/102115616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f350009bbd0 con 0x7f3504104b50 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f34f00a2d30 msgr2=0x7f34f00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f34f00a2d30 0x7f34f00a51f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f34f402a930 tx=0x7f34f4005fd0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 msgr2=0x7f3504197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 0x7f3504197f00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f350000c970 tx=0x7f350000ce30 comp rx=0 tx=0).stop 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 shutdown_connections 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f34f00a2d30 0x7f34f00a51f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3504107480 0x7f350419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504104b50 0x7f3504197f00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 --2- 172.21.15.16:0/102115616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3504102220 0x7f35041979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 >> 172.21.15.16:0/102115616 conn(0x7f35040fbc30 msgr2=0x7f3504102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 shutdown_connections 2024-04-25T01:38:38.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.709+0000 7f350c134640 1 -- 172.21.15.16:0/102115616 wait complete. 2024-04-25T01:38:38.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:38:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 -- 172.21.15.16:0/1347459680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 msgr2=0x7f90d4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 --2- 172.21.15.16:0/1347459680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d4108ae0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009ca0 tx=0x7f90cc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 -- 172.21.15.16:0/1347459680 shutdown_connections 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 --2- 172.21.15.16:0/1347459680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d4108ae0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 --2- 172.21.15.16:0/1347459680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d4101890 0x7f90d4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 --2- 172.21.15.16:0/1347459680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d41006a0 0x7f90d4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 -- 172.21.15.16:0/1347459680 >> 172.21.15.16:0/1347459680 conn(0x7f90d40fbc10 msgr2=0x7f90d40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 -- 172.21.15.16:0/1347459680 shutdown_connections 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 -- 172.21.15.16:0/1347459680 wait complete. 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 Processor -- start 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.826+0000 7f90db3b7640 1 -- start start 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d41006a0 0x7f90d406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d4101890 0x7f90d419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90d410c220 con 0x7f90d4102250 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90d410c0a0 con 0x7f90d41006a0 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f90d410c3a0 con 0x7f90d4101890 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d419ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53142/0 (socket says 172.21.15.16:53142) 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 -- 172.21.15.16:0/3926345676 learned_addr learned my addr 172.21.15.16:0/3926345676 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9153640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d41006a0 0x7f90d406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d8952640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d4101890 0x7f90d419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 -- 172.21.15.16:0/3926345676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d4101890 msgr2=0x7f90d419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d4101890 0x7f90d419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 -- 172.21.15.16:0/3926345676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d41006a0 msgr2=0x7f90d406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d41006a0 0x7f90d406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 -- 172.21.15.16:0/3926345676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d419f620 con 0x7f90d4102250 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d8952640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d4101890 0x7f90d419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90d9954640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d419ed10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f90cc031040 tx=0x7f90cc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc032ec0 con 0x7f90d4102250 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d419f850 con 0x7f90d4102250 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90cc0044f0 con 0x7f90d4102250 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc030bb0 con 0x7f90d4102250 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.827+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d419fda0 con 0x7f90d4102250 2024-04-25T01:38:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.828+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d41079c0 con 0x7f90d4102250 2024-04-25T01:38:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.830+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f90cc004040 con 0x7f90d4102250 2024-04-25T01:38:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.830+0000 7f90ca7fc640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2d30 0x7f90c00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.830+0000 7f90d9153640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2d30 0x7f90c00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.830+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f90cc0f0720 con 0x7f90d4102250 2024-04-25T01:38:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.831+0000 7f90d9153640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2d30 0x7f90c00a51f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f90d41016f0 tx=0x7f90d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:38.834+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f90cc0bd660 con 0x7f90d4102250 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.018+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f90d402d300 con 0x7f90d4102250 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.019+0000 7f90ca7fc640 1 -- 172.21.15.16:0/3926345676 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f90cc0bcf60 con 0x7f90d4102250 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:38:39.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2d30 msgr2=0x7f90c00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2d30 0x7f90c00a51f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f90d41016f0 tx=0x7f90d0005f90 comp rx=0 tx=0).stop 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 msgr2=0x7f90d419ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d419ed10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f90cc031040 tx=0x7f90cc003ba0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 shutdown_connections 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f90c00a2d30 0x7f90c00a51f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d4102250 0x7f90d419ed10 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90d4101890 0x7f90d419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 --2- 172.21.15.16:0/3926345676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d41006a0 0x7f90d406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 >> 172.21.15.16:0/3926345676 conn(0x7f90d40fbc10 msgr2=0x7f90d4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 shutdown_connections 2024-04-25T01:38:39.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.021+0000 7f90db3b7640 1 -- 172.21.15.16:0/3926345676 wait complete. 2024-04-25T01:38:39.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 -- 172.21.15.16:0/1207900844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c103350 msgr2=0x7fd53c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 --2- 172.21.15.16:0/1207900844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c103350 0x7fd53c109be0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd534009900 tx=0x7fd53402a410 comp rx=0 tx=0).stop 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 -- 172.21.15.16:0/1207900844 shutdown_connections 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 --2- 172.21.15.16:0/1207900844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c103350 0x7fd53c109be0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 --2- 172.21.15.16:0/1207900844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd53c102990 0x7fd53c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 --2- 172.21.15.16:0/1207900844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd53c1017a0 0x7fd53c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 -- 172.21.15.16:0/1207900844 >> 172.21.15.16:0/1207900844 conn(0x7fd53c0fcd10 msgr2=0x7fd53c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 -- 172.21.15.16:0/1207900844 shutdown_connections 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 -- 172.21.15.16:0/1207900844 wait complete. 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 Processor -- start 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.138+0000 7fd541dab640 1 -- start start 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 0x7fd53c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd53c102990 0x7fd53c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd53c103350 0x7fd53c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c10d2b0 con 0x7fd53c1017a0 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c10d130 con 0x7fd53c103350 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c10d430 con 0x7fd53c102990 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 0x7fd53c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 0x7fd53c197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53154/0 (socket says 172.21.15.16:53154) 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 -- 172.21.15.16:0/308070711 learned_addr learned my addr 172.21.15.16:0/308070711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53bfff640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd53c103350 0x7fd53c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 -- 172.21.15.16:0/308070711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd53c102990 msgr2=0x7fd53c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd53c102990 0x7fd53c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 -- 172.21.15.16:0/308070711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd53c103350 msgr2=0x7fd53c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd53c103350 0x7fd53c19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 -- 172.21.15.16:0/308070711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c19f570 con 0x7fd53c1017a0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd53b7fe640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 0x7fd53c197950 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00ebc0 tx=0x7fd52c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c003b70 con 0x7fd53c1017a0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd52c003d10 con 0x7fd53c1017a0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd53c19f860 con 0x7fd53c1017a0 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c019840 con 0x7fd53c1017a0 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.139+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd53c19fe00 con 0x7fd53c1017a0 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.140+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53c108b90 con 0x7fd53c1017a0 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.141+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd52c0199e0 con 0x7fd53c1017a0 2024-04-25T01:38:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.142+0000 7fd538ff9640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd5240a2c10 0x7fd5240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.142+0000 7fd53affd640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd5240a2c10 0x7fd5240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.142+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd52c0cfa90 con 0x7fd53c1017a0 2024-04-25T01:38:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.143+0000 7fd53affd640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd5240a2c10 0x7fd5240a50d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd530005fd0 tx=0x7fd530005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.146+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd52c09c970 con 0x7fd53c1017a0 2024-04-25T01:38:39.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:38 smithi156 ceph-mon[32631]: pgmap v696: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 58 MiB/s wr, 977 op/s 2024-04-25T01:38:39.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:38 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3846127253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:39.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:38 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/102115616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:38:39.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.331+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd53c062370 con 0x7fd53c1017a0 2024-04-25T01:38:39.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.332+0000 7fd538ff9640 1 -- 172.21.15.16:0/308070711 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fd52c09c270 con 0x7fd53c1017a0 2024-04-25T01:38:39.312 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:38:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd5240a2c10 msgr2=0x7fd5240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd5240a2c10 0x7fd5240a50d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd530005fd0 tx=0x7fd530005850 comp rx=0 tx=0).stop 2024-04-25T01:38:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 msgr2=0x7fd53c197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 0x7fd53c197950 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00ebc0 tx=0x7fd52c00c6a0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 shutdown_connections 2024-04-25T01:38:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd5240a2c10 0x7fd5240a50d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd53c103350 0x7fd53c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd53c102990 0x7fd53c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 --2- 172.21.15.16:0/308070711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd53c1017a0 0x7fd53c197950 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.333+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 >> 172.21.15.16:0/308070711 conn(0x7fd53c0fcd10 msgr2=0x7fd53c105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.334+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 shutdown_connections 2024-04-25T01:38:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.334+0000 7fd541dab640 1 -- 172.21.15.16:0/308070711 wait complete. 2024-04-25T01:38:39.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:38 smithi016 ceph-mon[29153]: pgmap v696: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 58 MiB/s wr, 977 op/s 2024-04-25T01:38:39.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:38 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3846127253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:39.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:38 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/102115616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:38:39.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4071994846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 msgr2=0x7f9f98109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4071994846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f98109910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009900 tx=0x7f9f9002a410 comp rx=0 tx=0).stop 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4071994846 shutdown_connections 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4071994846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f98109910 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4071994846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f98104bf0 0x7f9f98106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4071994846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f981022c0 0x7f9f981046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4071994846 >> 172.21.15.16:0/4071994846 conn(0x7f9f980fbc30 msgr2=0x7f9f980fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4071994846 shutdown_connections 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.449+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4071994846 wait complete. 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 Processor -- start 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 -- start start 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f981022c0 0x7f9f981979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f98104bf0 0x7f9f98197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f9819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9810d2f0 con 0x7f9f98107520 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9810d170 con 0x7f9f98104bf0 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9f0d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9810d470 con 0x7f9f981022c0 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9d671640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f9819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9d671640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f9819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53168/0 (socket says 172.21.15.16:53168) 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9d671640 1 -- 172.21.15.16:0/4012579396 learned_addr learned my addr 172.21.15.16:0/4012579396 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:38:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f97fff640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f98104bf0 0x7f9f98197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.450+0000 7f9f9ce70640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f981022c0 0x7f9f981979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9d671640 1 -- 172.21.15.16:0/4012579396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f981022c0 msgr2=0x7f9f981979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9d671640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f981022c0 0x7f9f981979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9d671640 1 -- 172.21.15.16:0/4012579396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f98104bf0 msgr2=0x7f9f98197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9d671640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f98104bf0 0x7f9f98197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9d671640 1 -- 172.21.15.16:0/4012579396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9819f5c0 con 0x7f9f98107520 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9d671640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f9819ef50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9f90005940 tx=0x7f9f90031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9003f030 con 0x7f9f98107520 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9819f850 con 0x7f9f98107520 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f90003aa0 con 0x7f9f98107520 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90035070 con 0x7f9f98107520 2024-04-25T01:38:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.451+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9819fd40 con 0x7f9f98107520 2024-04-25T01:38:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.452+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f980fd720 con 0x7f9f98107520 2024-04-25T01:38:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.454+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f9f90042030 con 0x7f9f98107520 2024-04-25T01:38:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.455+0000 7f9f95ffb640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9f840a2b90 0x7f9f840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.455+0000 7f9f9ce70640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9f840a2b90 0x7f9f840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.455+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9f900efa70 con 0x7f9f98107520 2024-04-25T01:38:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.455+0000 7f9f9ce70640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9f840a2b90 0x7f9f840a5050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9f88005fd0 tx=0x7f9f88009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.458+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9f900bc300 con 0x7f9f98107520 2024-04-25T01:38:39.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:38 smithi038 ceph-mon[32841]: pgmap v696: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 58 MiB/s wr, 977 op/s 2024-04-25T01:38:39.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:38 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3846127253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:38:39.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:38 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/102115616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:38:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.694+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f9f98062340 con 0x7f9f98107520 2024-04-25T01:38:39.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.702+0000 7f9f95ffb640 1 -- 172.21.15.16:0/4012579396 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+740371 (secure 0 0 0) 0x7f9f900c0600 con 0x7f9f98107520 2024-04-25T01:38:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:38:39.699 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:38:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 20, 2024-04-25T01:38:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 14152, 2024-04-25T01:38:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-04-25T01:38:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-04-25T01:38:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-25T01:38:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-25T01:38:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-04-25T01:38:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1051975299 2024-04-25T01:38:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-25T01:38:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-04-25T01:38:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1051975299 2024-04-25T01:38:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/1051975299", 2024-04-25T01:38:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-04-25T01:17:59.526839+0000", 2024-04-25T01:38:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T01:38:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-04-25T01:38:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-04-25T01:38:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24109, 2024-04-25T01:38:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-04-25T01:38:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-04-25T01:38:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-25T01:38:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-25T01:38:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:38:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-25T01:38:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-25T01:38:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-25T01:38:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-25T01:38:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-25T01:38:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-25T01:38:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:38:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:38:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-25T01:38:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-25T01:38:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-25T01:38:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-25T01:38:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-25T01:38:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-25T01:38:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-25T01:38:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-25T01:38:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:38:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:38:39.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:38:39.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:38:39.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:38:39.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.790 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.794 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.801 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:38:39.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.804 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.808 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.810 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:38:39.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:38:39.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:38:39.815 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.819 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-25T01:38:39.823 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.824 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.838 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.846 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.852 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:38:39.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.867 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-25T01:38:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-25T01:38:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:38:39.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-25T01:38:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-25T01:38:39.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.884 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-25T01:38:39.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.891 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-25T01:38:39.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.894 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.896 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:38:39.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:38:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:38:39.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.912 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.913 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.914 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.915 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-25T01:38:39.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-25T01:38:39.928 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-25T01:38:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-25T01:38:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:38:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:39.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-25T01:38:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:38:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-25T01:38:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-25T01:38:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:38:39.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-25T01:38:39.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-25T01:38:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-25T01:38:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:38:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.938 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.940 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-25T01:38:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.951 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.953 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.956 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.958 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:38:39.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.966 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.968 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.975 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-25T01:38:39.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:38:39.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-25T01:38:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-25T01:38:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.980 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.985 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.991 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:39.995 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:39.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:39.999 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.002 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.003 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.004 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.009 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.022 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.025 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.028 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.030 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.038 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.039 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.042 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.043 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.045 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.046 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.048 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.049 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:38:40.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-25T01:38:40.057 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.066 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.067 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-25T01:38:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.071 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.074 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.080 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.081 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.082 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.084 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.086 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.087 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.091 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.094 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.095 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.097 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.098 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-25T01:38:40.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.106 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.113 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:38:40.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.116 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.119 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.121 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.122 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.124 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.125 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.127 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.132 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-25T01:38:40.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.138 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.141 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.143 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-25T01:38:40.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.160 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:38:40.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.169 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.172 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.173 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.177 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-25T01:38:40.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-04-25T01:38:40.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-25T01:38:40.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:38:40.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-25T01:38:40.184 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.186 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.190 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-25T01:38:40.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-25T01:38:40.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-25T01:38:40.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 24113, 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.198 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.200 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.202 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-25T01:38:40.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.206 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-25T01:38:40.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.212 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:38:40.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:38:40.214 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:38:40.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:38:40.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-25T01:38:40.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.221 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.223 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.225 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.226 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.229 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.230 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.232 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.233 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.234 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.237 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-04-25T01:38:40.239 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-25T01:38:40.242 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:38:40.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-25T01:38:40.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.251 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:38:40.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:38:40.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.256 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.261 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.264 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:38:40.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.267 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.268 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:38:40.271 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:38:40.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:38:40.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-25T01:38:40.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:38:40.278 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.279 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.280 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-25T01:38:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-25T01:38:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.282 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:38:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:38:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.288 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-25T01:38:40.298 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.299 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:38:40.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-25T01:38:40.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-25T01:38:40.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.309 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.310 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:38:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-25T01:38:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-25T01:38:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-25T01:38:40.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-25T01:38:40.314 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-25T01:38:40.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-25T01:38:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.322 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.324 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.327 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.331 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.338 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.339 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.343 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:38:40.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.361 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-25T01:38:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-25T01:38:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.391 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-25T01:38:40.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.394 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.396 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.398 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.399 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-25T01:38:40.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.416 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.417 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-25T01:38:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-25T01:38:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-25T01:38:40.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-25T01:38:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-25T01:38:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-25T01:38:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.429 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:38:40.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.437 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.438 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.441 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.445 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.447 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.455 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.459 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.466 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.471 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.472 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.476 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.477 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.480 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.481 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.482 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.489 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.493 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.495 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.496 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.500 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.506 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.509 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.513 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.514 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.518 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-25T01:38:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.527 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.528 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.535 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.537 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:40.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-25T01:38:40.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.553 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.554 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.555 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-25T01:38:40.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.557 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.558 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.559 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.562 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.563 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.565 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.566 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.574 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.575 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.576 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.577 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.578 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.580 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.585 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.590 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.601 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.603 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.610 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.612 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.621 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.622 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.626 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.628 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.629 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:38:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.630 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.632 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.635 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.651 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.654 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.655 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.656 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.660 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:38:40.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:38:40.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:38:40.665 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-25T01:38:40.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.670 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.671 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.672 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-25T01:38:40.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-25T01:38:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-25T01:38:40.676 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:40.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "restful" 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.678 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.683 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-25T01:38:40.687 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-25T01:38:40.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-25T01:38:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.690 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-25T01:38:40.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:38:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.696 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:38:40.699 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:38:40.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:38:40.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.701 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-25T01:38:40.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-25T01:38:40.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.702 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-25T01:38:40.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-25T01:38:40.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-25T01:38:40.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:38:40.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.708 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.708 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.710 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.710 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.710 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.710 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.711 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.711 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-25T01:38:40.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-25T01:38:40.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.714 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.715 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-25T01:38:40.716 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:38:40.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:40.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-25T01:38:40.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:38:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.726 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.729 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-25T01:38:40.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.733 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.734 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-25T01:38:40.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-25T01:38:40.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.735 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.735 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.736 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.737 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:38:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.739 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-25T01:38:40.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.748 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:38:40.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:38:40.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-25T01:38:40.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-25T01:38:40.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-25T01:38:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:38:40.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-25T01:38:40.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-25T01:38:40.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-25T01:38:40.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.771 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:38:40.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.772 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.773 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.774 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.774 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.774 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.774 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.774 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.774 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.776 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-25T01:38:40.779 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.782 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.788 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.789 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.794 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.802 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.803 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.808 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.812 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.816 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.820 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.821 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.822 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:38:40.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:38:40.826 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:38:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:38:40.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:38:40.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:38:40.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:38:40.831 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:38:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:38:40.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:38:40.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:38:40.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.836 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:38:40.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:38:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:38:40.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:38:40.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.842 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.843 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.844 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.847 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.850 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.852 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.854 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.856 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.859 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-25T01:38:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.868 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.872 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.874 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.875 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-25T01:38:40.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-25T01:38:40.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.883 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.884 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-25T01:38:40.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-25T01:38:40.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.895 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.896 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.897 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-25T01:38:40.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-25T01:38:40.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.905 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-25T01:38:40.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-25T01:38:40.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.911 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.912 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.916 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.917 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:38:40.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.922 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-25T01:38:40.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.928 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.929 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:38:40.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.936 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.942 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.945 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.947 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.949 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.953 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.955 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.957 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.964 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.967 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.970 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.973 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.974 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.975 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.979 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.980 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.983 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-25T01:38:40.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-25T01:38:40.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:40.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:40.992 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:40.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:40.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-25T01:38:40.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:40.997 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:40.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.000 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.006 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-25T01:38:41.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-25T01:38:41.013 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.018 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.020 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-25T01:38:41.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:38:41.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-25T01:38:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:38:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:38:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.025 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.025 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-25T01:38:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-25T01:38:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-25T01:38:41.029 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.029 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.029 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-25T01:38:41.029 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-25T01:38:41.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.031 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.031 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-25T01:38:41.032 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-25T01:38:41.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.032 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.034 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.034 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.035 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.036 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.036 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.036 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.039 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.040 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.040 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.040 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.040 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.041 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.041 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.041 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.041 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.041 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.042 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.044 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-25T01:38:41.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-25T01:38:41.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.045 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-25T01:38:41.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.061 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.066 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.069 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.070 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-25T01:38:41.071 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-25T01:38:41.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-25T01:38:41.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.072 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.072 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:38:41.072 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-25T01:38:41.073 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.075 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.075 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.075 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.076 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.078 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.079 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.079 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.079 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.080 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.080 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.080 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.081 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.081 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.082 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.082 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.082 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.082 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.082 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.082 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-25T01:38:41.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-25T01:38:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-25T01:38:41.086 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-25T01:38:41.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-25T01:38:41.088 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-25T01:38:41.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.090 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.091 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-25T01:38:41.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-25T01:38:41.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.094 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.095 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.097 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.099 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.107 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-25T01:38:41.109 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:38:41.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.127 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.130 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.144 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.146 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.151 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-25T01:38:41.160 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.162 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.164 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.165 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-25T01:38:41.166 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.168 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-25T01:38:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-25T01:38:41.171 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-25T01:38:41.172 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-25T01:38:41.173 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-25T01:38:41.174 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "squid": [ 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 3, 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2861749009 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-04-25T01:38:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2156060995 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3919539389 2024-04-25T01:38:41.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4002106571 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-25T01:38:41.178 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9f840a2b90 msgr2=0x7f9f840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9f840a2b90 0x7f9f840a5050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9f88005fd0 tx=0x7f9f88009680 comp rx=0 tx=0).stop 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 msgr2=0x7f9f9819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f9819ef50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9f90005940 tx=0x7f9f90031a30 comp rx=0 tx=0).stop 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 shutdown_connections 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f9f840a2b90 0x7f9f840a5050 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f98107520 0x7f9f9819ef50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f98104bf0 0x7f9f98197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 --2- 172.21.15.16:0/4012579396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f981022c0 0x7f9f981979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 >> 172.21.15.16:0/4012579396 conn(0x7f9f980fbc30 msgr2=0x7f9f98102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 shutdown_connections 2024-04-25T01:38:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:38:39.708+0000 7f9f9f0d4640 1 -- 172.21.15.16:0/4012579396 wait complete. 2024-04-25T01:38:41.187 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T01:38:41.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T01:38:41.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:39 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3926345676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:38:41.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:39 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/308070711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:39 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4012579396' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:39 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3926345676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:39 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/308070711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:39 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/4012579396' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:39 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3926345676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:39 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/308070711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:38:41.188 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:39 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/4012579396' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:38:41.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:40 smithi016 ceph-mon[29153]: pgmap v697: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 54 MiB/s wr, 910 op/s 2024-04-25T01:38:41.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:40 smithi156 ceph-mon[32631]: pgmap v697: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 54 MiB/s wr, 910 op/s 2024-04-25T01:38:41.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:40 smithi038 ceph-mon[32841]: pgmap v697: 97 pgs: 97 active+clean; 10 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 54 MiB/s wr, 910 op/s 2024-04-25T01:38:41.603 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:38:43.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:43 smithi016 ceph-mon[29153]: pgmap v698: 97 pgs: 97 active+clean; 8.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 44 MiB/s wr, 775 op/s 2024-04-25T01:38:43.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:42 smithi156 ceph-mon[32631]: pgmap v698: 97 pgs: 97 active+clean; 8.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 44 MiB/s wr, 775 op/s 2024-04-25T01:38:43.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:43 smithi038 ceph-mon[32841]: pgmap v698: 97 pgs: 97 active+clean; 8.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 44 MiB/s wr, 775 op/s 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001605584, 2024-04-25T01:38:43.835 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000446427, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:38:43.836 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 224, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 219, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 755024, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 134944, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.037477071, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.013178023, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.017686652, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.008024399, 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:43.837 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 218, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.012194524, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000055938 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 755024, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 134944 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:38:43.838 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 5, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 169, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3200, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 107456, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.012489053, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.009072987, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001536671, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 168, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.007502629, 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000044658 2024-04-25T01:38:43.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3200, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 107456 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:38:43.840 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-25T01:38:43.841 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:38:43.842 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 149007, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:38:43.843 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-25T01:38:43.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 225, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 723383, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 225, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 225, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 723383, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:43.845 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:43.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:43.846 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:38:44.557 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-25T01:38:44.557 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-25T01:38:44.557 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-25T01:38:44.558 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-25T01:38:44.558 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T01:38:44.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T01:38:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:38:45.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:45 smithi156 ceph-mon[32631]: pgmap v699: 97 pgs: 97 active+clean; 3.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 36 MiB/s wr, 739 op/s 2024-04-25T01:38:45.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:45 smithi016 ceph-mon[29153]: pgmap v699: 97 pgs: 97 active+clean; 3.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 36 MiB/s wr, 739 op/s 2024-04-25T01:38:45.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:45 smithi038 ceph-mon[32841]: pgmap v699: 97 pgs: 97 active+clean; 3.5 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 36 MiB/s wr, 739 op/s 2024-04-25T01:38:46.707 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001391015, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:38:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000355994, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:38:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 171, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 160, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 107712, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.012242607, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.009039434, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001055598, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 169, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.007358677, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000043542 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 107712 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:38:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 234, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 227, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 759760, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 138752, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.037048862, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.012631768, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.017689373, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.008156673, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 227, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.009862472, 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000043447 2024-04-25T01:38:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 759760, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 138752 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:38:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:38:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:38:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 149007, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:38:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-25T01:38:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 231, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 724227, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 231, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 231, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 724227, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:46.720 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:38:47.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:47 smithi156 ceph-mon[32631]: pgmap v700: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 22 MiB/s wr, 537 op/s 2024-04-25T01:38:47.269 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:47 smithi038 ceph-mon[32841]: pgmap v700: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 22 MiB/s wr, 537 op/s 2024-04-25T01:38:47.325 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:47 smithi016 ceph-mon[29153]: pgmap v700: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 22 MiB/s wr, 537 op/s 2024-04-25T01:38:47.499 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T01:38:47.499 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T01:38:47.499 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-25T01:38:47.499 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-25T01:38:47.499 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T01:38:47.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T01:38:47.752 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:38:49.218 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:49 smithi016 ceph-mon[29153]: pgmap v701: 97 pgs: 97 active+clean; 181 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 19 MiB/s wr, 532 op/s 2024-04-25T01:38:49.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:49 smithi156 ceph-mon[32631]: pgmap v701: 97 pgs: 97 active+clean; 181 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 19 MiB/s wr, 532 op/s 2024-04-25T01:38:49.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:49 smithi038 ceph-mon[32841]: pgmap v701: 97 pgs: 97 active+clean; 181 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 19 MiB/s wr, 532 op/s 2024-04-25T01:38:49.484 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3040, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001738397, 2024-04-25T01:38:49.485 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000036609, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000635435, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000759449, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000253149 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.486 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 168, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 160, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 107184, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.012928277, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.009326147, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001723673, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:49.487 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 166, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.008434259, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000050808 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 107184 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:38:49.488 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 226, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 221, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 755568, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 136224, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.034653929, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.013480228, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.017502231, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.005182383, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 220, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.018250500, 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000082956 2024-04-25T01:38:49.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 755568, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 136224 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-25T01:38:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:38:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:38:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:38:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148984, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-25T01:38:49.494 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:38:49.495 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 227, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 723643, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 227, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 227, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 723643, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:49.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:49.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:49.497 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:38:50.292 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-25T01:38:50.292 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-25T01:38:50.293 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-25T01:38:50.293 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-25T01:38:50.293 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T01:38:50.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T01:38:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:38:51.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:51 smithi156 ceph-mon[32631]: pgmap v702: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 8.0 MiB/s wr, 356 op/s 2024-04-25T01:38:51.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:51 smithi016 ceph-mon[29153]: pgmap v702: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 8.0 MiB/s wr, 356 op/s 2024-04-25T01:38:51.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:51 smithi038 ceph-mon[32841]: pgmap v702: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 8.0 MiB/s wr, 356 op/s 2024-04-25T01:38:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:53 smithi016 ceph-mon[29153]: pgmap v703: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 923 KiB/s wr, 239 op/s 2024-04-25T01:38:53.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:53 smithi156 ceph-mon[32631]: pgmap v703: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 923 KiB/s wr, 239 op/s 2024-04-25T01:38:53.467 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:53 smithi038 ceph-mon[32841]: pgmap v703: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 923 KiB/s wr, 239 op/s 2024-04-25T01:38:53.558 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-25T01:38:53.558 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:38:53.558 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 233, 2024-04-25T01:38:53.558 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 228, 2024-04-25T01:38:53.558 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 757536, 2024-04-25T01:38:53.558 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 140224, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.039890381, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.013077679, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.020196772, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.008248566, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 227, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.014045450, 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000061874 2024-04-25T01:38:53.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 757536, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 140224 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 173, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 160, 2024-04-25T01:38:53.560 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 108064, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 3, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.012980936, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.009478906, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001407172, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 171, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.009714877, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000056812 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:53.561 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 108064 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3040, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-25T01:38:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.000549573, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000034244, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000468879, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000689898, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000229966 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:53.563 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:38:53.564 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:38:53.565 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-25T01:38:53.566 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148985, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-25T01:38:53.567 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-25T01:38:53.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 234, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 724617, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:53.569 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 234, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 234, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 724617, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-25T01:38:53.570 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-25T01:38:54.756 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T01:38:54.756 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T01:38:54.756 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-25T01:38:54.756 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-25T01:38:54.756 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T01:38:54.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T01:38:55.019 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:38:55.141 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:55 smithi038 ceph-mon[32841]: pgmap v704: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 194 op/s 2024-04-25T01:38:55.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:55 smithi156 ceph-mon[32631]: pgmap v704: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 194 op/s 2024-04-25T01:38:55.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:55 smithi016 ceph-mon[29153]: pgmap v704: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 194 op/s 2024-04-25T01:38:57.039 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:57 smithi038 ceph-mon[32841]: pgmap v705: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.5 KiB/s wr, 84 op/s 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2052, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 1885, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 613951, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 2449843, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 100, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 8, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.210789394, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.055655445, 2024-04-25T01:38:57.040 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.078471781, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.055889041, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1884, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.191821934, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000101816 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1077, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000208181, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000193 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 35392, 2024-04-25T01:38:57.041 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 1904 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2686, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2105, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1563334, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 997290, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 23, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.238271482, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.054802652, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.106688661, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.085262861, 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:57.042 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2104, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.541128133, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000257190 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1531, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000264830, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000172 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 840912, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 172176 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:38:57.043 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 1537, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 1596, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 460314, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 934190, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 8, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 6, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.140620622, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.044638902, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.044587616, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.041854831, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1594, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.143666776, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000090129 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 908, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000187285, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000206 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 384304 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-25T01:38:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 949, 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.213922942, 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000225419 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2466, 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.078903691, 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000031996 2024-04-25T01:38:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 100, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 100, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 100, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.166683898, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001666838 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-25T01:38:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 153, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 27, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 6, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 21, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 18, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 2, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 2, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 2, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 136, 2024-04-25T01:38:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 122, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-25T01:38:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 4, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 234, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-25T01:38:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 42, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 139, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 179, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 143, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 3, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 82 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 9, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-25T01:38:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 20, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 2, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 22, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 20, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-25T01:38:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 332, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 277, 2024-04-25T01:38:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 277, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 55, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 1, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 52, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 51, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 51, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 1, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 1, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 1, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4851422, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4971914, 2024-04-25T01:38:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 0, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 205, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.435945806, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002126564 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 0 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 25, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 45, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 20, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 20, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 22, 2024-04-25T01:38:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 2, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 27, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 66, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 39, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 2, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 36, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 34, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 56804, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 231804 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 151, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 100, 2024-04-25T01:38:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 19, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 22, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.029589879, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001344994 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 29, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.007038298, 2024-04-25T01:38:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000242699 2024-04-25T01:38:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-25T01:38:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-25T01:38:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.060 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.062 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-25T01:38:57.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 10, 2024-04-25T01:38:57.062 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.013755252, 2024-04-25T01:38:57.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001375525 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1, 2024-04-25T01:38:57.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000790952, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000790952 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 5, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.015633872, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.003126774 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.002230114, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001115057 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-25T01:38:57.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 4, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.050639456, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.012659864 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.001025010, 2024-04-25T01:38:57.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001025010 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.067 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 8, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.020751037, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002593879 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-25T01:38:57.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 18, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.025230028, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001401668 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 1, 2024-04-25T01:38:57.069 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 3, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 1, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 41, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 41, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 696, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:38:57.070 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-25T01:38:57.071 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-25T01:38:57.072 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 300457, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 125, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-25T01:38:57.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 7720, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 162, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 104668, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 865, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-25T01:38:57.074 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 3723, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 989132, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 33, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 3723, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3723, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 50.184092034, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.013479476 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3723, 2024-04-25T01:38:57.075 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 4106 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 3723, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 4, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 3719, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 131, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 44, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 4, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 209, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 138, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-25T01:38:57.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 3092, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 88, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:38:57.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-25T01:38:57.078 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 87, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 3, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 12, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 12, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 386, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 0, 2024-04-25T01:38:57.079 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 14 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 1, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 519, 2024-04-25T01:38:57.080 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 7, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 2580, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 20, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 6271, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 2047654, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 6271, 2024-04-25T01:38:57.081 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 6271, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2047654, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:38:57.082 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 3723, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 991152, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 3723, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 991152, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.083 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 3723, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 3723, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 3723, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 3723, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.084 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 20, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 2020, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 20, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:38:57.085 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 4, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2020, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x557bbcd64500": { 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 332, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 777610, 2024-04-25T01:38:57.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 332, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 205, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 777610, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:38:57.087 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-25T01:38:57.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:57 smithi156 ceph-mon[32631]: pgmap v705: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.5 KiB/s wr, 84 op/s 2024-04-25T01:38:57.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:57 smithi016 ceph-mon[29153]: pgmap v705: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.5 KiB/s wr, 84 op/s 2024-04-25T01:38:57.841 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-25T01:38:57.841 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-25T01:38:57.842 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=9 2024-04-25T01:38:57.842 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-04-25T01:38:57.842 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T01:38:57.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T01:38:58.099 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:38:59.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:38:59 smithi016 ceph-mon[29153]: pgmap v706: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 54 op/s 2024-04-25T01:38:59.446 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:38:59 smithi038 ceph-mon[32841]: pgmap v706: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 54 op/s 2024-04-25T01:38:59.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:38:59 smithi156 ceph-mon[32631]: pgmap v706: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 54 op/s 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:00 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 242, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 236, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 776866, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 141732, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.036277394, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.012325275, 2024-04-25T01:39:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.020372860, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005252357, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 235, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.008577269, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000036499 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 776032, 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 141408 2024-04-25T01:39:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 752, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 924, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 184928, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 619575, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 4, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.066822728, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.022945579, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.014100360, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.023319918, 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 922, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.030431639, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000033006 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 380752 2024-04-25T01:39:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 1648, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 1648, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 496416, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 523298, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.139835970, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.046853224, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.036277144, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.044971554, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1647, 2024-04-25T01:39:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.051388388, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000031201 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-25T01:39:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3963, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.101086784, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000025507 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 0, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 0, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-25T01:39:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 1, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 0, 2024-04-25T01:39:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-25T01:39:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:39:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 0 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-25T01:39:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-25T01:39:00.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 7, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 0, 2024-04-25T01:39:00.058 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 2, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4194304, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4196449, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 4196308, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.059 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 7 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 3, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 3, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 0, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 1, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 1, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 0, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 0, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 0, 2024-04-25T01:39:00.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 52416, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 215420 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-25T01:39:00.061 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.063 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.064 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-25T01:39:00.065 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.066 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-25T01:39:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:39:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 1, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 3, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 2, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 0, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 0, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 2945, 2024-04-25T01:39:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 189400, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 51, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:39:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 7720, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 162, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 8464, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 13, 2024-04-25T01:39:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 2396, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 21, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 3, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 2396, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2396, 2024-04-25T01:39:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 1.484972697, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000619771 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2396, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 2397 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 2396, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 2396, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 0, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 1567, 2024-04-25T01:39:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 827, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 1, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 0, 2024-04-25T01:39:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-25T01:39:00.081 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 87, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 2, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 5, 2024-04-25T01:39:00.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 5, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 0, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 2, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 0 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 0, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-25T01:39:00.083 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:39:00.084 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 2638, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1205346, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 2638, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2638, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1205346, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.085 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 2396, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 2150, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2396, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2150, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.086 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 2396, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 2396, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2396, 2024-04-25T01:39:00.087 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2396, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x55e5645366e0": { 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-25T01:39:00.089 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:00.090 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-25T01:39:00.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:39:00.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:00 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:39:00.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:00 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:00.843 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T01:39:00.843 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T01:39:00.843 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-25T01:39:00.843 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-25T01:39:00.843 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T01:39:00.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T01:39:01.097 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:39:01.267 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:01 smithi038 ceph-mon[32841]: pgmap v707: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T01:39:01.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:01 smithi016 ceph-mon[29153]: pgmap v707: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T01:39:01.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:01 smithi156 ceph-mon[32631]: pgmap v707: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T01:39:02.258 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:39:02.258 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:39:02.258 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:02 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:02.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:39:02.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:39:02.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:02 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:02.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:39:02.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:39:02.503 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:02 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' 2024-04-25T01:39:02.965 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 444, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 543, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 114869, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 230242, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 8, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.052561735, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.023095294, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.022907530, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004315870, 2024-04-25T01:39:02.966 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 546, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.034577297, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000063328 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 921, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000216822, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000235 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.967 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 292, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 284, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 791846, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 161289, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.038931578, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.012632730, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.023831101, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005631103, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 283, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.026268960, 2024-04-25T01:39:02.968 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000092823 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 16, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000001709, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000106 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 787472, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 148240 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 26, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 201, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 5280, 2024-04-25T01:39:02.969 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 434933, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 12, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 8, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.014216554, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.007420722, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.005017966, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000226364, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 195, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.014928406, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000076555 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 35, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000005970, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000170 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1744, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 383632 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-25T01:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 59, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.002436502, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000041296 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000007778, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000003889 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.972 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 7, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 4, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 4, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000701607, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000175401 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 3, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 11, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-25T01:39:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 16, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 0, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 16, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 13, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 8, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 3, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 18, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 12, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-25T01:39:02.974 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 5, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 4, 2024-04-25T01:39:02.975 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 234, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:39:02.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 1 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 12, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 8, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 10, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-25T01:39:02.977 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:39:02.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 7, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 7, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 2, 2024-04-25T01:39:02.979 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 2, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4194304, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4196449, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 0, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 6, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.015217640, 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002536273 2024-04-25T01:39:02.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 0 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 20, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 24, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 4, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 19, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 24, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 5, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 16, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 24, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 8, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-25T01:39:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 4, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 4, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 51348, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 223612 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 8, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 7, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 18, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.982 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000364473, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000182236 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.983 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000337134, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000168567 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.984 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.985 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-25T01:39:02.986 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.987 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.987 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.987 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-25T01:39:02.987 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.987 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-25T01:39:02.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.989 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.991 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:39:02.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 1, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 3, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 1, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-25T01:39:02.994 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 3, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 1, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 700, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:02.997 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 231975, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 87, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 176, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 2, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 7720, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 162, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:02.998 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 80693, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 333, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 45, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 12762, 2024-04-25T01:39:02.999 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 28, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 45, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 45, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.241995507, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.005377677 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 45, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 69 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 45, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 0, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 45, 2024-04-25T01:39:03.000 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 10, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 6, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 8, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-25T01:39:03.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 6, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 20, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-25T01:39:03.002 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-25T01:39:03.003 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 87, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 2, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 8, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 8, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 17, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 0, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 2 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.004 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 1, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-25T01:39:03.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 758, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 826463, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 758, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:03.006 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 758, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 826463, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:03.007 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 45, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 12762, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 45, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 12762, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:03.008 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 45, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 45, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 45, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 45, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x562cd3808500": { 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 7, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 2145, 2024-04-25T01:39:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 7, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 6, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2145, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:03.012 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-25T01:39:03.218 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:03 smithi038 ceph-mon[32841]: pgmap v708: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T01:39:03.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:03 smithi016 ceph-mon[29153]: pgmap v708: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T01:39:03.502 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:03 smithi156 ceph-mon[32631]: pgmap v708: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T01:39:03.815 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T01:39:03.815 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T01:39:03.815 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-25T01:39:03.815 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=12 2024-04-25T01:39:03.815 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T01:39:03.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T01:39:04.066 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:39:05.846 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:05 smithi038 ceph-mon[32841]: pgmap v709: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:05.846 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001222631, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000299021, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:05.847 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 4, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 178, 2024-04-25T01:39:05.848 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 2864, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 109040, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 3, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.009566625, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.006695990, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001235798, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 177, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.007767259, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000043882 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 2864, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 109040 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 237, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 232, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 758624, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 142784, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.035439463, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.011991346, 2024-04-25T01:39:05.850 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.020740645, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005351625, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 231, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.008724711, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000037769 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 758624, 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 142784 2024-04-25T01:39:05.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:05.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:05.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148984, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 30, 2024-04-25T01:39:05.854 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-25T01:39:05.855 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 238, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 725137, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 238, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-25T01:39:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 238, 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 725137, 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-25T01:39:05.857 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-25T01:39:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:05 smithi016 ceph-mon[29153]: pgmap v709: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:06.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:05 smithi156 ceph-mon[32631]: pgmap v709: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:06.561 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-25T01:39:06.561 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-25T01:39:06.561 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-25T01:39:06.561 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-25T01:39:06.561 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T01:39:06.562 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T01:39:06.815 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:39:07.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:07 smithi016 ceph-mon[29153]: pgmap v710: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:39:07.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:07 smithi038 ceph-mon[32841]: pgmap v710: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:39:08.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:07 smithi156 ceph-mon[32631]: pgmap v710: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-25T01:39:08.753 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1994, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1987, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 2073256, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 698712, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.169748876, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.053191430, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.054338069, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.052709886, 2024-04-25T01:39:08.754 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1986, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.060970705, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000030700 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 776464, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 142208 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.755 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 9, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 180, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 4856, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 387562, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.010222509, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.006678618, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.002054882, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000144377, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 179, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.014758480, 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000082449 2024-04-25T01:39:08.756 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3200, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 386224 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 810, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 812, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 199366, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 257587, 2024-04-25T01:39:08.757 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.065605410, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.021272445, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.014221999, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.024154676, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 810, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.028666504, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000035390 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.758 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4242, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.107923128, 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000025441 2024-04-25T01:39:08.759 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.760 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 54, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 4, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 50, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T01:39:08.761 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 47, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 2, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T01:39:08.762 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 4, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 234, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:39:08.763 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-04-25T01:39:08.764 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T01:39:08.765 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.766 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 332, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 52, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T01:39:08.767 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4851422, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4971914, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 4967277, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 332 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 16, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 36, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 20, 2024-04-25T01:39:08.768 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 11, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 13, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 2, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 54, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 57, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 3, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 51444, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215420 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-04-25T01:39:08.769 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.770 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T01:39:08.771 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.772 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.773 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T01:39:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.775 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T01:39:08.776 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 3, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 2, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 2, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 2945, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:08.782 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:08.783 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:08.784 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 189400, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 51, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:08.785 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 87756, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 572, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T01:39:08.786 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 2564, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 26, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 4, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 2564, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2564, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.222982008, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000476982 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T01:39:08.787 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2564, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2565 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 2564, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 2564, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 1634, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 927, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-04-25T01:39:08.788 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 2, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T01:39:08.789 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T01:39:08.790 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 2, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 5, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 5, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T01:39:08.791 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.792 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2809, 2024-04-25T01:39:08.793 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2009126, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2809, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2809, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2009126, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T01:39:08.794 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2564, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 777620, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2564, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 777620, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.795 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2564, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 2564, 2024-04-25T01:39:08.796 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2564, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2564, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55ea5e07e6e0": { 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T01:39:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.799 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:08.801 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T01:39:09.726 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T01:39:09.726 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-25T01:39:09.727 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-25T01:39:09.727 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-25T01:39:09.727 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T01:39:09.727 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T01:39:09.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:09 smithi016 ceph-mon[29153]: pgmap v711: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:39:09.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:09 smithi038 ceph-mon[32841]: pgmap v711: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:39:09.982 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:39:10.002 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:09 smithi156 ceph-mon[32631]: pgmap v711: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T01:39:11.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:11 smithi016 ceph-mon[29153]: pgmap v712: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:11.863 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:11 smithi156 ceph-mon[32631]: pgmap v712: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 2376, 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2378, 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 674180, 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 755575, 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T01:39:11.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.171750948, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.052232279, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.044167120, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.063279789, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2376, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.069796706, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000029375 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 3, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 177, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 716, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 386650, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.009448462, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.006660959, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.001425584, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000079621, 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:11.865 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 175, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.006746174, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000038549 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 385952 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:11.866 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 248, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 240, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 779884, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 143454, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 3, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.034907472, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.012156754, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.019528095, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.005312767, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 240, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.013354420, 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000055643 2024-04-25T01:39:11.867 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 779616, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 143136 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.868 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4011, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.095665709, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000023850 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.869 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T01:39:11.870 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 1, 2024-04-25T01:39:11.871 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T01:39:11.873 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:39:11.874 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-04-25T01:39:11.875 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 3, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4195385, 2024-04-25T01:39:11.878 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 4194892, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 3 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 3, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 3, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 0, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 1, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 1, 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 52528, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215420 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T01:39:11.882 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.883 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.885 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.885 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T01:39:11.885 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.885 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.886 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.887 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 1, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 0, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 1826, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:11.895 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 181383, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 48, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T01:39:11.896 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 8464, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 13, 2024-04-25T01:39:11.897 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 2379, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 2379, 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T01:39:11.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2379, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.253981593, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000527104 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2379, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2380 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 2379, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 2379, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T01:39:11.899 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 1542, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 835, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T01:39:11.900 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T01:39:11.901 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T01:39:11.902 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 2, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 4, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 4, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:39:11.903 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T01:39:11.904 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2623, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 1203162, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2623, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2623, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1203162, 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2379, 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1086, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2379, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1086, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2379, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 2379, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2379, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2379, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x5597158f86e0": { 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T01:39:11.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:11 smithi038 ceph-mon[32841]: pgmap v712: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:12.675 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T01:39:12.676 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T01:39:12.676 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-25T01:39:12.676 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-25T01:39:12.676 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T01:39:12.676 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T01:39:12.916 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:39:13.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:13 smithi156 ceph-mon[32631]: pgmap v713: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:39:13.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:13 smithi016 ceph-mon[29153]: pgmap v713: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:39:13.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:13 smithi038 ceph-mon[32841]: pgmap v713: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 701, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 805, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 899672, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 352257, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.083417396, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.030880506, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.044682384, 2024-04-25T01:39:14.700 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.010288121, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 804, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.040930538, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000050908 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 931, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000219171, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000235 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 789472, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 146048 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.701 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 29, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 202, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 6210, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 442945, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 11, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 6, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.013080897, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.007167400, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.004532857, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000230307, 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:14.702 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 200, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.034417265, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000172086 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 32, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000006804, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000212 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 387392 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:14.703 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 19, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 19, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 7203, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 13082, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.003318332, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.000241416, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.002531521, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000462874, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 18, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.015828051, 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000879336 2024-04-25T01:39:14.704 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.705 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 41, 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001517230, 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000037005 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-04-25T01:39:14.706 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000005289, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000002644 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 4, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 3, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000923529, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000307843 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 1, 2024-04-25T01:39:14.707 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 11, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 16, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 16, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 13, 2024-04-25T01:39:14.708 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 8, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 3, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 3, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 13, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 8, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 1, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 4, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T01:39:14.709 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 4, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 234, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:39:14.710 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.711 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 14, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 9, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 10, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T01:39:14.712 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 3, 2024-04-25T01:39:14.713 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 3, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 2, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4195385, 2024-04-25T01:39:14.714 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.003334356, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001667178 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 20, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 24, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 4, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 19, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 24, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 5, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 16, 2024-04-25T01:39:14.715 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 24, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 8, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 3, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 3, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 51404, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 223612 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 4, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 4, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 10, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T01:39:14.716 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000923529, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000307843 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.718 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.719 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T01:39:14.720 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.721 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.722 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.723 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.724 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.725 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.726 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.729 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.729 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.729 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.730 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T01:39:14.731 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.732 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 1, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 0, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 2, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 2, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 712, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.733 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:14.734 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:14.735 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 262923, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 88, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T01:39:14.736 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 81757, 2024-04-25T01:39:14.737 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 334, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 38, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 11512, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 28, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 38, 2024-04-25T01:39:14.738 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 38, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.240996419, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.006342011 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 38, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 61 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 38, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 0, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 38, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T01:39:14.739 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 10, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 2, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 6, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 6, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T01:39:14.740 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 20, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T01:39:14.741 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T01:39:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 2, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 10, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 10, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 16, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 1 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:39:14.743 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.744 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 745, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 829203, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 745, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 745, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 829203, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.745 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 38, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 11512, 2024-04-25T01:39:14.746 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 38, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 11512, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:14.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 38, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 38, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 38, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 38, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:14.748 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.749 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55cb64a0c500": { 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 3, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 1081, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 3, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1081, 2024-04-25T01:39:14.750 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:14.751 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:14.751 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:14.751 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:14.751 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.751 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:14.751 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T01:39:15.582 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-25T01:39:15.582 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-25T01:39:15.582 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-25T01:39:15.582 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=14 2024-04-25T01:39:15.582 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T01:39:15.585 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T01:39:15.824 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:39:15.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:15 smithi016 ceph-mon[29153]: pgmap v714: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:15.945 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:15 smithi156 ceph-mon[32631]: pgmap v714: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:15.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:15 smithi038 ceph-mon[32841]: pgmap v714: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:39:17.624 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:17 smithi156 ceph-mon[32631]: pgmap v715: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 0, 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2, 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 0, 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-04-25T01:39:17.624 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 0, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.001036205, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000230165, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:17.625 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 244, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 237, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 762480, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 145152, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 2, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.034647069, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.012233419, 2024-04-25T01:39:17.626 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.019708289, 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.005122430, 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 237, 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.010794702, 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000045547 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.627 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 762480, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 145152 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 179, 2024-04-25T01:39:17.628 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 160, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 109120, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 3, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.009627571, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.007214898, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000717327, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 177, 2024-04-25T01:39:17.629 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.008342222, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000047131 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 109120 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 0, 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T01:39:17.630 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:39:17.631 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:39:17.632 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 149047, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 31, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T01:39:17.633 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 928, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 2, 2024-04-25T01:39:17.634 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 241, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 725527, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 241, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T01:39:17.635 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 241, 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 725527, 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T01:39:17.636 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T01:39:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:17 smithi016 ceph-mon[29153]: pgmap v715: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T01:39:17.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:17 smithi038 ceph-mon[32841]: pgmap v715: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T01:39:18.321 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-25T01:39:18.321 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-25T01:39:18.321 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-25T01:39:18.321 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-25T01:39:18.322 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T01:39:18.332 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T01:39:18.332 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:39:18.333 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:39:18.362 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:39:18.362 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:39:18.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 -- 172.21.15.16:0/1378243665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 msgr2=0x7f1efc106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 --2- 172.21.15.16:0/1378243665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc106fa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009900 tx=0x7f1ee402a410 comp rx=0 tx=0).stop 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 -- 172.21.15.16:0/1378243665 shutdown_connections 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 --2- 172.21.15.16:0/1378243665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc1074e0 0x7f1efc1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 --2- 172.21.15.16:0/1378243665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc106fa0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 --2- 172.21.15.16:0/1378243665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc102280 0x7f1efc104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 -- 172.21.15.16:0/1378243665 >> 172.21.15.16:0/1378243665 conn(0x7f1efc0fbc30 msgr2=0x7f1efc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:18.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 -- 172.21.15.16:0/1378243665 shutdown_connections 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.595+0000 7f1f014f5640 1 -- 172.21.15.16:0/1378243665 wait complete. 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 Processor -- start 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 -- start start 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc102280 0x7f1efc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc1074e0 0x7f1efc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc10d2a0 con 0x7f1efc104bb0 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc10d120 con 0x7f1efc1074e0 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1f014f5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc10d420 con 0x7f1efc102280 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1ef3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1ef3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49272/0 (socket says 172.21.15.16:49272) 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1ef3fff640 1 -- 172.21.15.16:0/2194634889 learned_addr learned my addr 172.21.15.16:0/2194634889 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1efb7fe640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc1074e0 0x7f1efc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.596+0000 7f1efaffd640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc102280 0x7f1efc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef3fff640 1 -- 172.21.15.16:0/2194634889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc102280 msgr2=0x7f1efc197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef3fff640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc102280 0x7f1efc197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef3fff640 1 -- 172.21.15.16:0/2194634889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc1074e0 msgr2=0x7f1efc19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef3fff640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc1074e0 0x7f1efc19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef3fff640 1 -- 172.21.15.16:0/2194634889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc19f680 con 0x7f1efc104bb0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1efb7fe640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc1074e0 0x7f1efc19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1efaffd640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc102280 0x7f1efc197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef3fff640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc197fa0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40314e0 tx=0x7f1ee4031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee403f030 con 0x7f1efc104bb0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ee40044b0 con 0x7f1efc104bb0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4035070 con 0x7f1efc104bb0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc19f910 con 0x7f1efc104bb0 2024-04-25T01:39:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.597+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc19fdb0 con 0x7f1efc104bb0 2024-04-25T01:39:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.598+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1efc0fd720 con 0x7f1efc104bb0 2024-04-25T01:39:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.599+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1ee4042030 con 0x7f1efc104bb0 2024-04-25T01:39:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.600+0000 7f1ef8ff9640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ee80a2b90 0x7f1ee80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.600+0000 7f1efaffd640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ee80a2b90 0x7f1ee80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.600+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1ee40efa20 con 0x7f1efc104bb0 2024-04-25T01:39:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.601+0000 7f1efaffd640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ee80a2b90 0x7f1ee80a5050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009c20 tx=0x7f1eec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.604+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ee40bc900 con 0x7f1efc104bb0 2024-04-25T01:39:18.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.794+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1efc02d3c0 con 0x7f1efc104bb0 2024-04-25T01:39:18.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.796+0000 7f1ef8ff9640 1 -- 172.21.15.16:0/2194634889 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7f1ee40bc200 con 0x7f1efc104bb0 2024-04-25T01:39:18.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:39:18.779 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:39:18.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ee80a2b90 msgr2=0x7f1ee80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ee80a2b90 0x7f1ee80a5050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009c20 tx=0x7f1eec005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 msgr2=0x7f1efc197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc197fa0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40314e0 tx=0x7f1ee4031a30 comp rx=0 tx=0).stop 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 shutdown_connections 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f1ee80a2b90 0x7f1ee80a5050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1efc1074e0 0x7f1efc19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1efc104bb0 0x7f1efc197fa0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 --2- 172.21.15.16:0/2194634889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1efc102280 0x7f1efc197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 >> 172.21.15.16:0/2194634889 conn(0x7f1efc0fbc30 msgr2=0x7f1efc102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 shutdown_connections 2024-04-25T01:39:18.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.798+0000 7f1f014f5640 1 -- 172.21.15.16:0/2194634889 wait complete. 2024-04-25T01:39:18.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:39:18.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:39:18.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 -- 172.21.15.16:0/2513907157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 msgr2=0x7fbf98108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/2513907157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf98108ac0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009ca0 tx=0x7fbf9002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 -- 172.21.15.16:0/2513907157 shutdown_connections 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/2513907157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf98108ac0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/2513907157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf98101870 0x7fbf98101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/2513907157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf98100680 0x7fbf98100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 -- 172.21.15.16:0/2513907157 >> 172.21.15.16:0/2513907157 conn(0x7fbf980fbbf0 msgr2=0x7fbf980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 -- 172.21.15.16:0/2513907157 shutdown_connections 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.916+0000 7fbf9f89e640 1 -- 172.21.15.16:0/2513907157 wait complete. 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 Processor -- start 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 -- start start 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf98100680 0x7fbf981979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf98101870 0x7fbf98197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf9819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810c190 con 0x7fbf98102230 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810c010 con 0x7fbf98101870 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9f89e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810c310 con 0x7fbf98100680 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf9819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf9819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49286/0 (socket says 172.21.15.16:49286) 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 -- 172.21.15.16:0/1717456596 learned_addr learned my addr 172.21.15.16:0/1717456596 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9ce39640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf98101870 0x7fbf98197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9d63a640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf98100680 0x7fbf981979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 -- 172.21.15.16:0/1717456596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf98100680 msgr2=0x7fbf981979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf98100680 0x7fbf981979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 -- 172.21.15.16:0/1717456596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf98101870 msgr2=0x7fbf98197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf98101870 0x7fbf98197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.917+0000 7fbf9de3b640 1 -- 172.21.15.16:0/1717456596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9819f5e0 con 0x7fbf98102230 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.918+0000 7fbf9de3b640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf9819ef70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbf90006010 tx=0x7fbf90003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.918+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf90004040 con 0x7fbf98102230 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.918+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf90004690 con 0x7fbf98102230 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.918+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9819f870 con 0x7fbf98102230 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.918+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf90030af0 con 0x7fbf98102230 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.918+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9819fdc0 con 0x7fbf98102230 2024-04-25T01:39:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.919+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf981a0020 con 0x7fbf98102230 2024-04-25T01:39:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.920+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fbf90030c90 con 0x7fbf98102230 2024-04-25T01:39:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.921+0000 7fbf8e7fc640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf840a2b90 0x7fbf840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.921+0000 7fbf9d63a640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf840a2b90 0x7fbf840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.921+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbf900f0900 con 0x7fbf98102230 2024-04-25T01:39:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.921+0000 7fbf9d63a640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf840a2b90 0x7fbf840a5050 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbf981016d0 tx=0x7fbf94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:18.924+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf900bd960 con 0x7fbf98102230 2024-04-25T01:39:19.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.110+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbf981a03b0 con 0x7fbf98102230 2024-04-25T01:39:19.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.111+0000 7fbf8e7fc640 1 -- 172.21.15.16:0/1717456596 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fbf900bd260 con 0x7fbf98102230 2024-04-25T01:39:19.093 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf840a2b90 msgr2=0x7fbf840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf840a2b90 0x7fbf840a5050 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbf981016d0 tx=0x7fbf94005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 msgr2=0x7fbf9819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf9819ef70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbf90006010 tx=0x7fbf90003ba0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 shutdown_connections 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fbf840a2b90 0x7fbf840a5050 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf98102230 0x7fbf9819ef70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf98101870 0x7fbf98197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 --2- 172.21.15.16:0/1717456596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf98100680 0x7fbf981979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 >> 172.21.15.16:0/1717456596 conn(0x7fbf980fbbf0 msgr2=0x7fbf980fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:19.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 shutdown_connections 2024-04-25T01:39:19.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.113+0000 7fbf9f89e640 1 -- 172.21.15.16:0/1717456596 wait complete. 2024-04-25T01:39:19.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 -- 172.21.15.16:0/1071378094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc102230 msgr2=0x7f12dc108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 --2- 172.21.15.16:0/1071378094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc102230 0x7f12dc108ac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ca0 tx=0x7f12d402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 -- 172.21.15.16:0/1071378094 shutdown_connections 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 --2- 172.21.15.16:0/1071378094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc102230 0x7f12dc108ac0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 --2- 172.21.15.16:0/1071378094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12dc101870 0x7f12dc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 --2- 172.21.15.16:0/1071378094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc100680 0x7f12dc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 -- 172.21.15.16:0/1071378094 >> 172.21.15.16:0/1071378094 conn(0x7f12dc0fbbf0 msgr2=0x7f12dc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 -- 172.21.15.16:0/1071378094 shutdown_connections 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.230+0000 7f12e12df640 1 -- 172.21.15.16:0/1071378094 wait complete. 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 Processor -- start 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 -- start start 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 0x7f12dc1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc101870 0x7f12dc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12dc102230 0x7f12dc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc10c1b0 con 0x7f12dc100680 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc10c030 con 0x7f12dc102230 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.231+0000 7f12e12df640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc10c330 con 0x7f12dc101870 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 0x7f12dc1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 0x7f12dc1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49312/0 (socket says 172.21.15.16:49312) 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 -- 172.21.15.16:0/340909625 learned_addr learned my addr 172.21.15.16:0/340909625 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12db7fe640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12dc102230 0x7f12dc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12da7fc640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc101870 0x7f12dc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 -- 172.21.15.16:0/340909625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc101870 msgr2=0x7f12dc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc101870 0x7f12dc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 -- 172.21.15.16:0/340909625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12dc102230 msgr2=0x7f12dc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12dc102230 0x7f12dc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 -- 172.21.15.16:0/340909625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12dc19f5d0 con 0x7f12dc100680 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12da7fc640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc101870 0x7f12dc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12daffd640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 0x7f12dc1979b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00e9c0 tx=0x7f12cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc00cd80 con 0x7f12dc100680 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12dc19f8c0 con 0x7f12dc100680 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12cc0035f0 con 0x7f12dc100680 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc010690 con 0x7f12dc100680 2024-04-25T01:39:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.232+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12dc19fdc0 con 0x7f12dc100680 2024-04-25T01:39:19.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.233+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12dc1a0020 con 0x7f12dc100680 2024-04-25T01:39:19.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.234+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f12cc0037a0 con 0x7f12dc100680 2024-04-25T01:39:19.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.235+0000 7f12d3fff640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f12c80a2b90 0x7f12c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.235+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f12cc0cf640 con 0x7f12dc100680 2024-04-25T01:39:19.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.235+0000 7f12da7fc640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f12c80a2b90 0x7f12c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.235+0000 7f12da7fc640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f12c80a2b90 0x7f12c80a5050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f12dc198ec0 tx=0x7f12c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:19.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.239+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f12cc09c580 con 0x7f12dc100680 2024-04-25T01:39:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.427+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f12dc1a01f0 con 0x7f12dc100680 2024-04-25T01:39:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.428+0000 7f12d3fff640 1 -- 172.21.15.16:0/340909625 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f12cc09be80 con 0x7f12dc100680 2024-04-25T01:39:19.410 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f12c80a2b90 msgr2=0x7f12c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f12c80a2b90 0x7f12c80a5050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f12dc198ec0 tx=0x7f12c4005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 msgr2=0x7f12dc1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 0x7f12dc1979b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00e9c0 tx=0x7f12cc00ee80 comp rx=0 tx=0).stop 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 shutdown_connections 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f12c80a2b90 0x7f12c80a5050 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12dc102230 0x7f12dc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12dc101870 0x7f12dc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 --2- 172.21.15.16:0/340909625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12dc100680 0x7f12dc1979b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 >> 172.21.15.16:0/340909625 conn(0x7f12dc0fbbf0 msgr2=0x7f12dc0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 shutdown_connections 2024-04-25T01:39:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.430+0000 7f12e12df640 1 -- 172.21.15.16:0/340909625 wait complete. 2024-04-25T01:39:19.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:39:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/1041253007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f681017a0 msgr2=0x7f8f68101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/1041253007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f681017a0 0x7f8f68101bc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009900 tx=0x7f8f5802a410 comp rx=0 tx=0).stop 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/1041253007 shutdown_connections 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/1041253007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f68103350 0x7f8f68109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/1041253007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 0x7f8f68102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/1041253007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f681017a0 0x7f8f68101bc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/1041253007 >> 172.21.15.16:0/1041253007 conn(0x7f8f680fcd10 msgr2=0x7f8f680ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/1041253007 shutdown_connections 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.549+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/1041253007 wait complete. 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 Processor -- start 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 -- start start 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f681017a0 0x7f8f681979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 0x7f8f68197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 0x7f8f6819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810d2d0 con 0x7f8f68103350 2024-04-25T01:39:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810d150 con 0x7f8f68102990 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f6dcf7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810d450 con 0x7f8f681017a0 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 0x7f8f6819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 0x7f8f6819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49332/0 (socket says 172.21.15.16:49332) 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f67fff640 1 -- 172.21.15.16:0/563301450 learned_addr learned my addr 172.21.15.16:0/563301450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.550+0000 7f8f677fe640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f681017a0 0x7f8f681979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f66ffd640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 0x7f8f68197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f67fff640 1 -- 172.21.15.16:0/563301450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f681017a0 msgr2=0x7f8f681979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f67fff640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f681017a0 0x7f8f681979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f67fff640 1 -- 172.21.15.16:0/563301450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 msgr2=0x7f8f68197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f67fff640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 0x7f8f68197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f67fff640 1 -- 172.21.15.16:0/563301450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6819f610 con 0x7f8f68103350 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f66ffd640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 0x7f8f68197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f67fff640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 0x7f8f6819efa0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000eaa0 tx=0x7f8f6000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6000cdb0 con 0x7f8f68103350 2024-04-25T01:39:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f60005020 con 0x7f8f68103350 2024-04-25T01:39:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6819f900 con 0x7f8f68103350 2024-04-25T01:39:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f600116d0 con 0x7f8f68103350 2024-04-25T01:39:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.551+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f6819fe50 con 0x7f8f68103350 2024-04-25T01:39:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.552+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f68108be0 con 0x7f8f68103350 2024-04-25T01:39:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.553+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8f600051d0 con 0x7f8f68103350 2024-04-25T01:39:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.554+0000 7f8f64ff9640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f500a2b90 0x7f8f500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.554+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8f600cf220 con 0x7f8f68103350 2024-04-25T01:39:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.554+0000 7f8f677fe640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f500a2b90 0x7f8f500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.554+0000 7f8f677fe640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f500a2b90 0x7f8f500a5050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8f58004770 tx=0x7f8f58000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.558+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8f60016030 con 0x7f8f68103350 2024-04-25T01:39:19.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.752+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8f68062340 con 0x7f8f68103350 2024-04-25T01:39:19.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.753+0000 7f8f64ff9640 1 -- 172.21.15.16:0/563301450 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f8f6009c100 con 0x7f8f68103350 2024-04-25T01:39:19.735 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f500a2b90 msgr2=0x7f8f500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f500a2b90 0x7f8f500a5050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8f58004770 tx=0x7f8f58000bc0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 msgr2=0x7f8f6819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 0x7f8f6819efa0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000eaa0 tx=0x7f8f6000ef60 comp rx=0 tx=0).stop 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 shutdown_connections 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f8f500a2b90 0x7f8f500a5050 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f68103350 0x7f8f6819efa0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68102990 0x7f8f68197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 --2- 172.21.15.16:0/563301450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f681017a0 0x7f8f681979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 >> 172.21.15.16:0/563301450 conn(0x7f8f680fcd10 msgr2=0x7f8f68105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 shutdown_connections 2024-04-25T01:39:19.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.755+0000 7f8f6dcf7640 1 -- 172.21.15.16:0/563301450 wait complete. 2024-04-25T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:19 smithi016 ceph-mon[29153]: pgmap v716: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:19 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2194634889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:19 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1717456596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:39:19.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:19 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/340909625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:39:19.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 -- 172.21.15.16:0/4075052096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c102990 msgr2=0x7f073c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 --2- 172.21.15.16:0/4075052096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c102990 0x7f073c102e10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f072c009900 tx=0x7f072c02a410 comp rx=0 tx=0).stop 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 -- 172.21.15.16:0/4075052096 shutdown_connections 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 --2- 172.21.15.16:0/4075052096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f073c103350 0x7f073c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 --2- 172.21.15.16:0/4075052096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c102990 0x7f073c102e10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 --2- 172.21.15.16:0/4075052096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f073c1017a0 0x7f073c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 -- 172.21.15.16:0/4075052096 >> 172.21.15.16:0/4075052096 conn(0x7f073c0fcd10 msgr2=0x7f073c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 -- 172.21.15.16:0/4075052096 shutdown_connections 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.870+0000 7f0743493640 1 -- 172.21.15.16:0/4075052096 wait complete. 2024-04-25T01:39:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 Processor -- start 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 -- start start 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 0x7f073c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f073c102990 0x7f073c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f073c103350 0x7f073c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f073c10d2b0 con 0x7f073c1017a0 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f073c10d130 con 0x7f073c103350 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0743493640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f073c10d430 con 0x7f073c102990 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 0x7f073c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 0x7f073c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49342/0 (socket says 172.21.15.16:49342) 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 -- 172.21.15.16:0/4160114314 learned_addr learned my addr 172.21.15.16:0/4160114314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0740a2e640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f073c102990 0x7f073c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f0741a30640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f073c103350 0x7f073c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 -- 172.21.15.16:0/4160114314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f073c102990 msgr2=0x7f073c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f073c102990 0x7f073c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 -- 172.21.15.16:0/4160114314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f073c103350 msgr2=0x7f073c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f073c103350 0x7f073c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.871+0000 7f074122f640 1 -- 172.21.15.16:0/4160114314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c19f620 con 0x7f073c1017a0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.872+0000 7f074122f640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 0x7f073c197a00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f073800d950 tx=0x7f073800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.872+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738017020 con 0x7f073c1017a0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.872+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0738009c30 con 0x7f073c1017a0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.872+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073c19f910 con 0x7f073c1017a0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.872+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073800f6b0 con 0x7f073c1017a0 2024-04-25T01:39:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.872+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073c19fe30 con 0x7f073c1017a0 2024-04-25T01:39:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.873+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f073c0689b0 con 0x7f073c1017a0 2024-04-25T01:39:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.874+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f073801a020 con 0x7f073c1017a0 2024-04-25T01:39:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.874+0000 7f07327fc640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07280a2b90 0x7f07280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.875+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f07380ced10 con 0x7f073c1017a0 2024-04-25T01:39:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.875+0000 7f0740a2e640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07280a2b90 0x7f07280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.875+0000 7f0740a2e640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07280a2b90 0x7f07280a5050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f072c006010 tx=0x7f072c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:19.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:19.879+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0738009de0 con 0x7f073c1017a0 2024-04-25T01:39:19.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:19 smithi038 ceph-mon[32841]: pgmap v716: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T01:39:19.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:19 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2194634889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:19.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:19 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1717456596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:39:19.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:19 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/340909625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:39:20.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:19 smithi156 ceph-mon[32631]: pgmap v716: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T01:39:20.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:19 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2194634889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:20.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:19 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1717456596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:39:20.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:19 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/340909625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:39:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.079+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f073c062340 con 0x7f073c1017a0 2024-04-25T01:39:20.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.079+0000 7f07327fc640 1 -- 172.21.15.16:0/4160114314 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+591 (secure 0 0 0) 0x7f073809bc50 con 0x7f073c1017a0 2024-04-25T01:39:20.061 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1190188268 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4006719933 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2073759739 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1190188268 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1590801448 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1662737541 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2428618530 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1662737541 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2674684320 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/153993658 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2024-04-25T01:39:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07280a2b90 msgr2=0x7f07280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07280a2b90 0x7f07280a5050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f072c006010 tx=0x7f072c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 msgr2=0x7f073c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 0x7f073c197a00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f073800d950 tx=0x7f073800de10 comp rx=0 tx=0).stop 2024-04-25T01:39:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 shutdown_connections 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f07280a2b90 0x7f07280a5050 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f073c103350 0x7f073c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f073c102990 0x7f073c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 --2- 172.21.15.16:0/4160114314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c1017a0 0x7f073c197a00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 >> 172.21.15.16:0/4160114314 conn(0x7f073c0fcd10 msgr2=0x7f073c105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 shutdown_connections 2024-04-25T01:39:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.082+0000 7f0743493640 1 -- 172.21.15.16:0/4160114314 wait complete. 2024-04-25T01:39:20.075 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-25T01:39:20.076 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:39:20.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 -- 172.21.15.16:0/1725401134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c1006a0 msgr2=0x7fd77c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 --2- 172.21.15.16:0/1725401134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c1006a0 0x7fd77c100ac0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009900 tx=0x7fd76c02a410 comp rx=0 tx=0).stop 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 -- 172.21.15.16:0/1725401134 shutdown_connections 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 --2- 172.21.15.16:0/1725401134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd77c102250 0x7fd77c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 --2- 172.21.15.16:0/1725401134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c101890 0x7fd77c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 --2- 172.21.15.16:0/1725401134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c1006a0 0x7fd77c100ac0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 -- 172.21.15.16:0/1725401134 >> 172.21.15.16:0/1725401134 conn(0x7fd77c0fbc30 msgr2=0x7fd77c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 -- 172.21.15.16:0/1725401134 shutdown_connections 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.253+0000 7fd78099e640 1 -- 172.21.15.16:0/1725401134 wait complete. 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 Processor -- start 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 -- start start 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c1006a0 0x7fd77c104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd77c101890 0x7fd77c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 0x7fd77c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10c2a0 con 0x7fd77c102250 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10c120 con 0x7fd77c101890 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd78099e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10c420 con 0x7fd77c1006a0 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd77ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 0x7fd77c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd77ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 0x7fd77c1031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49362/0 (socket says 172.21.15.16:49362) 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd77ad9d640 1 -- 172.21.15.16:0/3183662647 learned_addr learned my addr 172.21.15.16:0/3183662647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:20.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd773fff640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd77c101890 0x7fd77c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.254+0000 7fd77a59c640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c1006a0 0x7fd77c104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77ad9d640 1 -- 172.21.15.16:0/3183662647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c1006a0 msgr2=0x7fd77c104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77ad9d640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c1006a0 0x7fd77c104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77ad9d640 1 -- 172.21.15.16:0/3183662647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd77c101890 msgr2=0x7fd77c102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77ad9d640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd77c101890 0x7fd77c102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77ad9d640 1 -- 172.21.15.16:0/3183662647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c103a40 con 0x7fd77c102250 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77a59c640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c1006a0 0x7fd77c104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd77ad9d640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 0x7fd77c1031b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b970 tx=0x7fd77400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774019070 con 0x7fd77c102250 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd774014440 con 0x7fd77c102250 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c1a1cd0 con 0x7fd77c102250 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7740075d0 con 0x7fd77c102250 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.255+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c1a21d0 con 0x7fd77c102250 2024-04-25T01:39:20.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.256+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c1a24c0 con 0x7fd77c102250 2024-04-25T01:39:20.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.257+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd77401c070 con 0x7fd77c102250 2024-04-25T01:39:20.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.258+0000 7fd7737fe640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7680a2c10 0x7fd7680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.258+0000 7fd77a59c640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7680a2c10 0x7fd7680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.258+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd7740ce680 con 0x7fd77c102250 2024-04-25T01:39:20.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.258+0000 7fd77a59c640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7680a2c10 0x7fd7680a50d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd76c006010 tx=0x7fd76c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:20.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.262+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd77409b5c0 con 0x7fd77c102250 2024-04-25T01:39:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.448+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd77c1a2850 con 0x7fd77c102250 2024-04-25T01:39:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.450+0000 7fd7737fe640 1 -- 172.21.15.16:0/3183662647 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7fd77409aec0 con 0x7fd77c102250 2024-04-25T01:39:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:39:20.432 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:39:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7680a2c10 msgr2=0x7fd7680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7680a2c10 0x7fd7680a50d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd76c006010 tx=0x7fd76c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 msgr2=0x7fd77c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 0x7fd77c1031b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b970 tx=0x7fd77400be30 comp rx=0 tx=0).stop 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 shutdown_connections 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd7680a2c10 0x7fd7680a50d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c102250 0x7fd77c1031b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd77c101890 0x7fd77c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 --2- 172.21.15.16:0/3183662647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd77c1006a0 0x7fd77c104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 >> 172.21.15.16:0/3183662647 conn(0x7fd77c0fbc30 msgr2=0x7fd77c069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 shutdown_connections 2024-04-25T01:39:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.452+0000 7fd78099e640 1 -- 172.21.15.16:0/3183662647 wait complete. 2024-04-25T01:39:20.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:20:28.050972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:25:14.398788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:25:15.720822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:39:20.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.570+0000 7f60b2f09640 1 -- 172.21.15.16:0/219828713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac103350 msgr2=0x7f60ac109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.570+0000 7f60b2f09640 1 --2- 172.21.15.16:0/219828713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac103350 0x7f60ac109be0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f609c009900 tx=0x7f609c02a410 comp rx=0 tx=0).stop 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- 172.21.15.16:0/219828713 shutdown_connections 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 --2- 172.21.15.16:0/219828713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac103350 0x7f60ac109be0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 --2- 172.21.15.16:0/219828713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ac102990 0x7f60ac102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 --2- 172.21.15.16:0/219828713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac1017a0 0x7f60ac101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- 172.21.15.16:0/219828713 >> 172.21.15.16:0/219828713 conn(0x7f60ac0fcd10 msgr2=0x7f60ac0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- 172.21.15.16:0/219828713 shutdown_connections 2024-04-25T01:39:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- 172.21.15.16:0/219828713 wait complete. 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 Processor -- start 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- start start 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ac1017a0 0x7f60ac197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 0x7f60ac197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac103350 0x7f60ac19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac10d2d0 con 0x7f60ac102990 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac10d150 con 0x7f60ac1017a0 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60b2f09640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac10d450 con 0x7f60ac103350 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 0x7f60ac197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60abfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 0x7f60ac197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49366/0 (socket says 172.21.15.16:49366) 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.571+0000 7f60abfff640 1 -- 172.21.15.16:0/482855768 learned_addr learned my addr 172.21.15.16:0/482855768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60b0ca5640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ac1017a0 0x7f60ac197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60b14a6640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac103350 0x7f60ac19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60abfff640 1 -- 172.21.15.16:0/482855768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac103350 msgr2=0x7f60ac19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60abfff640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac103350 0x7f60ac19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60abfff640 1 -- 172.21.15.16:0/482855768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ac1017a0 msgr2=0x7f60ac197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60abfff640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ac1017a0 0x7f60ac197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60abfff640 1 -- 172.21.15.16:0/482855768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac19f530 con 0x7f60ac102990 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60abfff640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 0x7f60ac197e50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f60a000c920 tx=0x7f60a000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0007c70 con 0x7f60ac102990 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ac19f820 con 0x7f60ac102990 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60a0007e10 con 0x7f60ac102990 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0018820 con 0x7f60ac102990 2024-04-25T01:39:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.572+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ac19fd70 con 0x7f60ac102990 2024-04-25T01:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.573+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ac108ac0 con 0x7f60ac102990 2024-04-25T01:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.574+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f60a00189c0 con 0x7f60ac102990 2024-04-25T01:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.575+0000 7f60a9ffb640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f60940a2b90 0x7f60940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.575+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f60a00cf1f0 con 0x7f60ac102990 2024-04-25T01:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.575+0000 7f60b0ca5640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f60940a2b90 0x7f60940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.576+0000 7f60b0ca5640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f60940a2b90 0x7f60940a5050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f60ac1027f0 tx=0x7f60a4009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.579+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60a0014030 con 0x7f60ac102990 2024-04-25T01:39:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.764+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f60ac062340 con 0x7f60ac102990 2024-04-25T01:39:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.764+0000 7f60a9ffb640 1 -- 172.21.15.16:0/482855768 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f60a009c0d0 con 0x7f60ac102990 2024-04-25T01:39:20.748 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f60940a2b90 msgr2=0x7f60940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f60940a2b90 0x7f60940a5050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f60ac1027f0 tx=0x7f60a4009680 comp rx=0 tx=0).stop 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 msgr2=0x7f60ac197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 0x7f60ac197e50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f60a000c920 tx=0x7f60a000cde0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 shutdown_connections 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f60940a2b90 0x7f60940a5050 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60ac103350 0x7f60ac19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60ac102990 0x7f60ac197e50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.766+0000 7f60b2f09640 1 --2- 172.21.15.16:0/482855768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ac1017a0 0x7f60ac197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.767+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 >> 172.21.15.16:0/482855768 conn(0x7f60ac0fcd10 msgr2=0x7f60ac105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:20.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.767+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 shutdown_connections 2024-04-25T01:39:20.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.767+0000 7f60b2f09640 1 -- 172.21.15.16:0/482855768 wait complete. 2024-04-25T01:39:20.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:20 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/563301450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:39:20.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:20 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/4160114314' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:39:20.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:20 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3183662647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:20.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 -- 172.21.15.16:0/1205023865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8102250 msgr2=0x7fd1b8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/1205023865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8102250 0x7fd1b8108ae0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009ca0 tx=0x7fd1b002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 -- 172.21.15.16:0/1205023865 shutdown_connections 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/1205023865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8102250 0x7fd1b8108ae0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/1205023865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1b8101890 0x7fd1b8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/1205023865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 0x7fd1b8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 -- 172.21.15.16:0/1205023865 >> 172.21.15.16:0/1205023865 conn(0x7fd1b80fbc30 msgr2=0x7fd1b80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 -- 172.21.15.16:0/1205023865 shutdown_connections 2024-04-25T01:39:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.883+0000 7fd1bf03f640 1 -- 172.21.15.16:0/1205023865 wait complete. 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 Processor -- start 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 -- start start 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 0x7fd1b8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 0x7fd1b8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1b8102250 0x7fd1b819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b810c270 con 0x7fd1b8101890 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b810c0f0 con 0x7fd1b81006a0 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bf03f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b810c3f0 con 0x7fd1b8102250 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 0x7fd1b8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 0x7fd1b8197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49390/0 (socket says 172.21.15.16:49390) 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1b7fff640 1 -- 172.21.15.16:0/499069389 learned_addr learned my addr 172.21.15.16:0/499069389 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bd5dc640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1b8102250 0x7fd1b819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.884+0000 7fd1bcddb640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 0x7fd1b8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b7fff640 1 -- 172.21.15.16:0/499069389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1b8102250 msgr2=0x7fd1b819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b7fff640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1b8102250 0x7fd1b819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b7fff640 1 -- 172.21.15.16:0/499069389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 msgr2=0x7fd1b8197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b7fff640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 0x7fd1b8197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b7fff640 1 -- 172.21.15.16:0/499069389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b819f630 con 0x7fd1b8101890 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1bcddb640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 0x7fd1b8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b7fff640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 0x7fd1b8197f50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400d770 tx=0x7fd1a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4017070 con 0x7fd1b8101890 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b819f920 con 0x7fd1b8101890 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a40054d0 con 0x7fd1b8101890 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4004cf0 con 0x7fd1b8101890 2024-04-25T01:39:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.885+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b819fe70 con 0x7fd1b8101890 2024-04-25T01:39:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.886+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b81a00a0 con 0x7fd1b8101890 2024-04-25T01:39:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.887+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd1a4004e90 con 0x7fd1b8101890 2024-04-25T01:39:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.888+0000 7fd1b5ffb640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1a80a2b90 0x7fd1a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.888+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd1a40ceb30 con 0x7fd1b8101890 2024-04-25T01:39:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.888+0000 7fd1bcddb640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1a80a2b90 0x7fd1a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.889+0000 7fd1bcddb640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1a80a2b90 0x7fd1a80a5050 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd1b81016f0 tx=0x7fd1ac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:20.892+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1a409ba10 con 0x7fd1b8101890 2024-04-25T01:39:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:20 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/563301450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:39:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:20 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/4160114314' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:39:20.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:20 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3183662647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:21.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:20 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/563301450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:39:21.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:20 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/4160114314' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:39:21.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:20 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3183662647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.077+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd1b81a0390 con 0x7fd1b8101890 2024-04-25T01:39:21.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.077+0000 7fd1b5ffb640 1 -- 172.21.15.16:0/499069389 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fd1a409b310 con 0x7fd1b8101890 2024-04-25T01:39:21.061 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1a80a2b90 msgr2=0x7fd1a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1a80a2b90 0x7fd1a80a5050 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd1b81016f0 tx=0x7fd1ac008040 comp rx=0 tx=0).stop 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 msgr2=0x7fd1b8197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 0x7fd1b8197f50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400d770 tx=0x7fd1a400dc30 comp rx=0 tx=0).stop 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 shutdown_connections 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fd1a80a2b90 0x7fd1a80a5050 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1b8102250 0x7fd1b819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1b8101890 0x7fd1b8197f50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 --2- 172.21.15.16:0/499069389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b81006a0 0x7fd1b8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.079+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 >> 172.21.15.16:0/499069389 conn(0x7fd1b80fbc30 msgr2=0x7fd1b80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:21.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.080+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 shutdown_connections 2024-04-25T01:39:21.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.080+0000 7fd1bf03f640 1 -- 172.21.15.16:0/499069389 wait complete. 2024-04-25T01:39:21.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 -- 172.21.15.16:0/7973857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2538101890 msgr2=0x7f2538101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 --2- 172.21.15.16:0/7973857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2538101890 0x7f2538101d10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f252c009900 tx=0x7f252c02a410 comp rx=0 tx=0).stop 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 -- 172.21.15.16:0/7973857 shutdown_connections 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 --2- 172.21.15.16:0/7973857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2538102250 0x7f2538108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 --2- 172.21.15.16:0/7973857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2538101890 0x7f2538101d10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 --2- 172.21.15.16:0/7973857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25381006a0 0x7f2538100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 -- 172.21.15.16:0/7973857 >> 172.21.15.16:0/7973857 conn(0x7f25380fbc30 msgr2=0x7f25380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 -- 172.21.15.16:0/7973857 shutdown_connections 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 -- 172.21.15.16:0/7973857 wait complete. 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 Processor -- start 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.197+0000 7f253d838640 1 -- start start 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253d838640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 0x7f2538197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253d838640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2538101890 0x7f2538197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253d838640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2538102250 0x7f253819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253d838640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f253810d350 con 0x7f25381006a0 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253d838640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f253810d1d0 con 0x7f2538101890 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253d838640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f253810d4d0 con 0x7f2538102250 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 0x7f2538197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 0x7f2538197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49418/0 (socket says 172.21.15.16:49418) 2024-04-25T01:39:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 -- 172.21.15.16:0/823967072 learned_addr learned my addr 172.21.15.16:0/823967072 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f25367fc640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2538101890 0x7f2538197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f25377fe640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2538102250 0x7f253819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 -- 172.21.15.16:0/823967072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2538102250 msgr2=0x7f253819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2538102250 0x7f253819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 -- 172.21.15.16:0/823967072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2538101890 msgr2=0x7f2538197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2538101890 0x7f2538197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 -- 172.21.15.16:0/823967072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253819f610 con 0x7f25381006a0 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f2536ffd640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 0x7f2538197a90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f252800e9c0 tx=0x7f252800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252800cd80 con 0x7f25381006a0 2024-04-25T01:39:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.198+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25280035f0 con 0x7f25381006a0 2024-04-25T01:39:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.199+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253819f900 con 0x7f25381006a0 2024-04-25T01:39:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.199+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528010690 con 0x7f25381006a0 2024-04-25T01:39:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.199+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253819fe20 con 0x7f25381006a0 2024-04-25T01:39:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.200+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25381079e0 con 0x7f25381006a0 2024-04-25T01:39:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.201+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f25280037a0 con 0x7f25381006a0 2024-04-25T01:39:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.201+0000 7f253c836640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25200a2b90 0x7f25200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.201+0000 7f25367fc640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25200a2b90 0x7f25200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.201+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f25280cf4b0 con 0x7f25381006a0 2024-04-25T01:39:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.202+0000 7f25367fc640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25200a2b90 0x7f25200a5050 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f252c0314e0 tx=0x7f252c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.206+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f252809c3f0 con 0x7f25381006a0 2024-04-25T01:39:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.391+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2538062340 con 0x7f25381006a0 2024-04-25T01:39:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.391+0000 7f253c836640 1 -- 172.21.15.16:0/823967072 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f252809bcf0 con 0x7f25381006a0 2024-04-25T01:39:21.374 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25200a2b90 msgr2=0x7f25200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25200a2b90 0x7f25200a5050 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f252c0314e0 tx=0x7f252c005a20 comp rx=0 tx=0).stop 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 msgr2=0x7f2538197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 0x7f2538197a90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f252800e9c0 tx=0x7f252800ee80 comp rx=0 tx=0).stop 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 shutdown_connections 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f25200a2b90 0x7f25200a5050 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2538102250 0x7f253819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2538101890 0x7f2538197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 --2- 172.21.15.16:0/823967072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25381006a0 0x7f2538197a90 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 >> 172.21.15.16:0/823967072 conn(0x7f25380fbc30 msgr2=0x7f2538104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 shutdown_connections 2024-04-25T01:39:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.393+0000 7f253d838640 1 -- 172.21.15.16:0/823967072 wait complete. 2024-04-25T01:39:21.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:39:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 -- 172.21.15.16:0/2113517617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 msgr2=0x7f0908102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 --2- 172.21.15.16:0/2113517617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908102e10 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009900 tx=0x7f08fc02a410 comp rx=0 tx=0).stop 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 -- 172.21.15.16:0/2113517617 shutdown_connections 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 --2- 172.21.15.16:0/2113517617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0908103350 0x7f0908109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 --2- 172.21.15.16:0/2113517617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908102e10 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 --2- 172.21.15.16:0/2113517617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09081017a0 0x7f0908101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 -- 172.21.15.16:0/2113517617 >> 172.21.15.16:0/2113517617 conn(0x7f09080fcd10 msgr2=0x7f09080ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.508+0000 7f090d8bd640 1 -- 172.21.15.16:0/2113517617 shutdown_connections 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 -- 172.21.15.16:0/2113517617 wait complete. 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 Processor -- start 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 -- start start 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09081017a0 0x7f09081979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0908103350 0x7f090819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f090810d2b0 con 0x7f0908102990 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f090810d130 con 0x7f09081017a0 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f090d8bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f090810d430 con 0x7f0908103350 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.509+0000 7f09067fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49440/0 (socket says 172.21.15.16:49440) 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 -- 172.21.15.16:0/3011009629 learned_addr learned my addr 172.21.15.16:0/3011009629 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09077fe640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0908103350 0x7f090819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f0906ffd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09081017a0 0x7f09081979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 -- 172.21.15.16:0/3011009629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0908103350 msgr2=0x7f090819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0908103350 0x7f090819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 -- 172.21.15.16:0/3011009629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09081017a0 msgr2=0x7f09081979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09081017a0 0x7f09081979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 -- 172.21.15.16:0/3011009629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090819f5d0 con 0x7f0908102990 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f0906ffd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09081017a0 0x7f09081979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09077fe640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0908103350 0x7f090819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f09067fc640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908197ef0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f08fc004770 tx=0x7f08fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc03f030 con 0x7f0908102990 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090819f860 con 0x7f0908102990 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08fc003cb0 con 0x7f0908102990 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc035070 con 0x7f0908102990 2024-04-25T01:39:21.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.510+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090819fd50 con 0x7f0908102990 2024-04-25T01:39:21.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.511+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0908108b90 con 0x7f0908102990 2024-04-25T01:39:21.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.512+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f08fc042030 con 0x7f0908102990 2024-04-25T01:39:21.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.513+0000 7f090c8bb640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f08f00a2b90 0x7f08f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.513+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f08fc0efaf0 con 0x7f0908102990 2024-04-25T01:39:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.513+0000 7f0906ffd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f08f00a2b90 0x7f08f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.513+0000 7f0906ffd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f08f00a2b90 0x7f08f00a5050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f09081027f0 tx=0x7f08f8005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.517+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f08fc0bca30 con 0x7f0908102990 2024-04-25T01:39:21.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.718+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f090802d170 con 0x7f0908102990 2024-04-25T01:39:21.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.718+0000 7f090c8bb640 1 -- 172.21.15.16:0/3011009629 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+591 (secure 0 0 0) 0x7f08fc0bc330 con 0x7f0908102990 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1190188268 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4006719933 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2073759739 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1190188268 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1590801448 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1662737541 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2428618530 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1662737541 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2674684320 2024-04-26T01:17:31.392949+0000 2024-04-25T01:39:21.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/153993658 2024-04-26T01:17:59.526723+0000 2024-04-25T01:39:21.703 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.720+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f08f00a2b90 msgr2=0x7f08f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.720+0000 7f090d8bd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f08f00a2b90 0x7f08f00a5050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f09081027f0 tx=0x7f08f8005e70 comp rx=0 tx=0).stop 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.720+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 msgr2=0x7f0908197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.720+0000 7f090d8bd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908197ef0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f08fc004770 tx=0x7f08fc0042f0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.720+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 shutdown_connections 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.720+0000 7f090d8bd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f08f00a2b90 0x7f08f00a5050 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.721+0000 7f090d8bd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0908103350 0x7f090819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.721+0000 7f090d8bd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0908102990 0x7f0908197ef0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.721+0000 7f090d8bd640 1 --2- 172.21.15.16:0/3011009629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09081017a0 0x7f09081979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.721+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 >> 172.21.15.16:0/3011009629 conn(0x7f09080fcd10 msgr2=0x7f0908105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.721+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 shutdown_connections 2024-04-25T01:39:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:21.721+0000 7f090d8bd640 1 -- 172.21.15.16:0/3011009629 wait complete. 2024-04-25T01:39:21.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:21 smithi016 ceph-mon[29153]: pgmap v717: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T01:39:21.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:21 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/482855768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:39:21.714 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:21 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/499069389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:39:21.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:21 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/823967072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:39:21.715 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-25T01:39:21.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:21.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T01:39:21.765 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:39:21.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[80756]: fuse finished with error 0 and tester_r 0 2024-04-25T01:39:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:21 smithi038 ceph-mon[32841]: pgmap v717: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T01:39:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:21 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/482855768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:39:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:21 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/499069389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:39:21.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:21 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/823967072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:39:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:21 smithi156 ceph-mon[32631]: pgmap v717: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T01:39:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:21 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/482855768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:39:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:21 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/499069389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:39:22.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:21 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/823967072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:39:22.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:22 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3011009629' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:39:23.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:22 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3011009629' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:39:23.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:22 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3011009629' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:39:23.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:23 smithi038 ceph-mon[32841]: pgmap v718: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:39:24.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:23 smithi156 ceph-mon[32631]: pgmap v718: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:39:24.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:23 smithi016 ceph-mon[29153]: pgmap v718: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:39:25.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:25 smithi038 ceph-mon[32841]: pgmap v719: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:39:26.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:25 smithi156 ceph-mon[32631]: pgmap v719: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:39:26.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:25 smithi016 ceph-mon[29153]: pgmap v719: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:39:27.866 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:39:27.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:27.867 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:39:27.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:27.920 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-25T01:39:27.943 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:39:27.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:27.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:39:27.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:27 smithi038 ceph-mon[32841]: pgmap v720: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:39:28.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:27 smithi156 ceph-mon[32631]: pgmap v720: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:39:28.008 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:27 smithi016 ceph-mon[29153]: pgmap v720: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:39:28.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:28.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-25T01:39:28.101 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T01:39:28.112 INFO:tasks.cephadm:Teardown begin 2024-04-25T01:39:28.112 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:39:28.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 -- 172.21.15.16:0/3721115127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c1017b0 msgr2=0x7f120c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 --2- 172.21.15.16:0/3721115127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c1017b0 0x7f120c101bd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1208009900 tx=0x7f120802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 -- 172.21.15.16:0/3721115127 shutdown_connections 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 --2- 172.21.15.16:0/3721115127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f120c103360 0x7f120c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 --2- 172.21.15.16:0/3721115127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f120c1029a0 0x7f120c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 --2- 172.21.15.16:0/3721115127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c1017b0 0x7f120c101bd0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 -- 172.21.15.16:0/3721115127 >> 172.21.15.16:0/3721115127 conn(0x7f120c0fcd40 msgr2=0x7f120c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 -- 172.21.15.16:0/3721115127 shutdown_connections 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.237+0000 7f12133a2640 1 -- 172.21.15.16:0/3721115127 wait complete. 2024-04-25T01:39:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 Processor -- start 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 -- start start 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f120c1017b0 0x7f120c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f120c1029a0 0x7f120c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 0x7f120c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f120c10d2d0 con 0x7f120c103360 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f120c10d150 con 0x7f120c1017b0 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f12133a2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f120c10d450 con 0x7f120c1029a0 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f121193f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 0x7f120c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f121193f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 0x7f120c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51722/0 (socket says 172.21.15.16:51722) 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f121193f640 1 -- 172.21.15.16:0/2946265598 learned_addr learned my addr 172.21.15.16:0/2946265598 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.238+0000 7f121093d640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f120c1029a0 0x7f120c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121113e640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f120c1017b0 0x7f120c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121193f640 1 -- 172.21.15.16:0/2946265598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f120c1029a0 msgr2=0x7f120c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121193f640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f120c1029a0 0x7f120c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121193f640 1 -- 172.21.15.16:0/2946265598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f120c1017b0 msgr2=0x7f120c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121193f640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f120c1017b0 0x7f120c0754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121193f640 1 -- 172.21.15.16:0/2946265598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c074900 con 0x7f120c103360 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f121193f640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 0x7f120c074070 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f120400cce0 tx=0x7f120400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120400f450 con 0x7f120c103360 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120400faa0 con 0x7f120c103360 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120c1a1ac0 con 0x7f120c103360 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1204018b70 con 0x7f120c103360 2024-04-25T01:39:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.239+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120c1a2010 con 0x7f120c103360 2024-04-25T01:39:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.240+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f120c108bf0 con 0x7f120c103360 2024-04-25T01:39:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.241+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f1204018d10 con 0x7f120c103360 2024-04-25T01:39:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.242+0000 7f12027fc640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11f80a2c10 0x7f11f80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.242+0000 7f121113e640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11f80a2c10 0x7f11f80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.242+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f12040cf590 con 0x7f120c103360 2024-04-25T01:39:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.242+0000 7f121113e640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11f80a2c10 0x7f11f80a50d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f120802a910 tx=0x7f1208000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:28.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.245+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1204014030 con 0x7f120c103360 2024-04-25T01:39:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.471+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f120c02d170 con 0x7f120c103360 2024-04-25T01:39:28.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.471+0000 7f12027fc640 1 -- 172.21.15.16:0/2946265598 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7f120409c580 con 0x7f120c103360 2024-04-25T01:39:28.454 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:39:28.454 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:27:21.048344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24476,24499]},"id":1}]} 2024-04-25T01:39:28.454 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11f80a2c10 msgr2=0x7f11f80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11f80a2c10 0x7f11f80a50d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f120802a910 tx=0x7f1208000bc0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 msgr2=0x7f120c074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 0x7f120c074070 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f120400cce0 tx=0x7f120400a620 comp rx=0 tx=0).stop 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 shutdown_connections 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f11f80a2c10 0x7f11f80a50d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120c103360 0x7f120c074070 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f120c1029a0 0x7f120c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 --2- 172.21.15.16:0/2946265598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f120c1017b0 0x7f120c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 >> 172.21.15.16:0/2946265598 conn(0x7f120c0fcd40 msgr2=0x7f120c105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 shutdown_connections 2024-04-25T01:39:28.457 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.473+0000 7f12133a2640 1 -- 172.21.15.16:0/2946265598 wait complete. 2024-04-25T01:39:28.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 -- 172.21.15.16:0/2409179813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c102240 msgr2=0x7f829c108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/2409179813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c102240 0x7f829c108ad0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8294009ca0 tx=0x7f829402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 -- 172.21.15.16:0/2409179813 shutdown_connections 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/2409179813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c102240 0x7f829c108ad0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/2409179813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 0x7f829c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/2409179813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f829c100690 0x7f829c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 -- 172.21.15.16:0/2409179813 >> 172.21.15.16:0/2409179813 conn(0x7f829c0fbc20 msgr2=0x7f829c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 -- 172.21.15.16:0/2409179813 shutdown_connections 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.588+0000 7f82a1bb0640 1 -- 172.21.15.16:0/2409179813 wait complete. 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 Processor -- start 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 -- start start 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 0x7f829c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 0x7f829c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f829c102240 0x7f829c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10d2a0 con 0x7f829c100690 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10d120 con 0x7f829c102240 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f82a1bb0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10d420 con 0x7f829c101880 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f829b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 0x7f829c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f829b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 0x7f829c197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51742/0 (socket says 172.21.15.16:51742) 2024-04-25T01:39:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.589+0000 7f829b7fe640 1 -- 172.21.15.16:0/1156200151 learned_addr learned my addr 172.21.15.16:0/1156200151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829affd640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 0x7f829c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829b7fe640 1 -- 172.21.15.16:0/1156200151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 msgr2=0x7f829c197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829b7fe640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 0x7f829c197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829b7fe640 1 -- 172.21.15.16:0/1156200151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f829c102240 msgr2=0x7f829c19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829b7fe640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f829c102240 0x7f829c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829b7fe640 1 -- 172.21.15.16:0/1156200151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c19f590 con 0x7f829c100690 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829affd640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 0x7f829c197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f829b7fe640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 0x7f829c197970 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f828c00d950 tx=0x7f828c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c017020 con 0x7f829c100690 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f828c009c30 con 0x7f829c100690 2024-04-25T01:39:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c19f880 con 0x7f829c100690 2024-04-25T01:39:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c00f6b0 con 0x7f829c100690 2024-04-25T01:39:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.590+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c19fdd0 con 0x7f829c100690 2024-04-25T01:39:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.591+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829c1079b0 con 0x7f829c100690 2024-04-25T01:39:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.592+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f828c01a020 con 0x7f829c100690 2024-04-25T01:39:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.593+0000 7f8298ff9640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f82840a2b90 0x7f82840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.593+0000 7f829affd640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f82840a2b90 0x7f82840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.593+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f828c0cefc0 con 0x7f829c100690 2024-04-25T01:39:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.593+0000 7f829affd640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f82840a2b90 0x7f82840a5050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f829c198e80 tx=0x7f8290005cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:28.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.597+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f828c021020 con 0x7f829c100690 2024-04-25T01:39:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.824+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f829c02d390 con 0x7f829c100690 2024-04-25T01:39:28.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.825+0000 7f8298ff9640 1 -- 172.21.15.16:0/1156200151 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7f828c009de0 con 0x7f829c100690 2024-04-25T01:39:28.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:39:28.808 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/690842306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":690842306},{"type":"v1","addr":"172.21.15.16:6835","nonce":690842306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2204481226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2204481226},{"type":"v1","addr":"172.21.15.16:6837","nonce":2204481226}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/4015716085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":4015716085},{"type":"v1","addr":"172.21.15.16:6839","nonce":4015716085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/3747963498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":3747963498},{"type":"v1","addr":"172.21.15.16:6841","nonce":3747963498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1629519202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1629519202},{"type":"v1","addr":"172.21.15.156:6833","nonce":1629519202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/3531176332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":3531176332},{"type":"v1","addr":"172.21.15.38:6833","nonce":3531176332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:25:16.327985+0000","modified":"2024-04-25T01:27:21.048344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/2628472521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2628472521},{"type":"v1","addr":"172.21.15.38:6835","nonce":2628472521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/2186765223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2186765223},{"type":"v1","addr":"172.21.15.156:6835","nonce":2186765223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/2504948433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2504948433},{"type":"v1","addr":"172.21.15.156:6837","nonce":2504948433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.38:6837/401186797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":401186797},{"type":"v1","addr":"172.21.15.38:6837","nonce":401186797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/1428073868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1428073868},{"type":"v1","addr":"172.21.15.156:6839","nonce":1428073868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/2790744598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":2790744598},{"type":"v1","addr":"172.21.15.38:6839","nonce":2790744598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24476,24499]},"id":1}]} 2024-04-25T01:39:28.808 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f82840a2b90 msgr2=0x7f82840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f82840a2b90 0x7f82840a5050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f829c198e80 tx=0x7f8290005cb0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 msgr2=0x7f829c197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 0x7f829c197970 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f828c00d950 tx=0x7f828c00de10 comp rx=0 tx=0).stop 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 shutdown_connections 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f82840a2b90 0x7f82840a5050 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f829c102240 0x7f829c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c101880 0x7f829c197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 --2- 172.21.15.16:0/1156200151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f829c100690 0x7f829c197970 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 >> 172.21.15.16:0/1156200151 conn(0x7f829c0fbc20 msgr2=0x7f829c104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 shutdown_connections 2024-04-25T01:39:28.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.827+0000 7f82a1bb0640 1 -- 172.21.15.16:0/1156200151 wait complete. 2024-04-25T01:39:28.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 -- 172.21.15.16:0/2358338129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1024102250 msgr2=0x7f1024108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 --2- 172.21.15.16:0/2358338129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1024102250 0x7f1024108ae0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f101c009ca0 tx=0x7f101c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 -- 172.21.15.16:0/2358338129 shutdown_connections 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 --2- 172.21.15.16:0/2358338129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1024102250 0x7f1024108ae0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 --2- 172.21.15.16:0/2358338129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 0x7f1024101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 --2- 172.21.15.16:0/2358338129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10241006a0 0x7f1024100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 -- 172.21.15.16:0/2358338129 >> 172.21.15.16:0/2358338129 conn(0x7f10240fbc30 msgr2=0x7f10240fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 -- 172.21.15.16:0/2358338129 shutdown_connections 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 -- 172.21.15.16:0/2358338129 wait complete. 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 Processor -- start 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.942+0000 7f102a31b640 1 -- start start 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f102a31b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 0x7f10241968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f102a31b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 0x7f1024196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f102a31b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1024102250 0x7f102419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f102a31b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f102410c280 con 0x7f10241006a0 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f102a31b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f102410c100 con 0x7f1024102250 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f102a31b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f102410c400 con 0x7f1024101890 2024-04-25T01:39:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 0x7f10241968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 0x7f10241968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51770/0 (socket says 172.21.15.16:51770) 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 -- 172.21.15.16:0/2784569450 learned_addr learned my addr 172.21.15.16:0/2784569450 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f10237fe640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 0x7f1024196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f10288b8640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1024102250 0x7f102419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 -- 172.21.15.16:0/2784569450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 msgr2=0x7f1024196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 0x7f1024196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 -- 172.21.15.16:0/2784569450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1024102250 msgr2=0x7f102419de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1024102250 0x7f102419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 -- 172.21.15.16:0/2784569450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102419e4c0 con 0x7f10241006a0 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f10237fe640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 0x7f1024196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.943+0000 7f1023fff640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 0x7f10241968a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f101400c970 tx=0x7f101400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.944+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014007c10 con 0x7f10241006a0 2024-04-25T01:39:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.944+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1014007db0 con 0x7f10241006a0 2024-04-25T01:39:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.944+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102419e7b0 con 0x7f10241006a0 2024-04-25T01:39:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.944+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101400f870 con 0x7f10241006a0 2024-04-25T01:39:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.944+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102419ed00 con 0x7f10241006a0 2024-04-25T01:39:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.945+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1024107960 con 0x7f10241006a0 2024-04-25T01:39:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.946+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f101400fa10 con 0x7f10241006a0 2024-04-25T01:39:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.947+0000 7f10217fa640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f100c0a2b90 0x7f100c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.947+0000 7f10237fe640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f100c0a2b90 0x7f100c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.947+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f10140cf770 con 0x7f10241006a0 2024-04-25T01:39:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.947+0000 7f10237fe640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f100c0a2b90 0x7f100c0a5050 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1024197db0 tx=0x7f1018008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:28.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:28.950+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f101409c770 con 0x7f10241006a0 2024-04-25T01:39:28.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:28 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2946265598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:28 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2946265598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:28 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2946265598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.138+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1024062370 con 0x7f10241006a0 2024-04-25T01:39:29.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.140+0000 7f10217fa640 1 -- 172.21.15.16:0/2784569450 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22723 (secure 0 0 0) 0x7f101409c070 con 0x7f10241006a0 2024-04-25T01:39:29.123 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-25T01:39:29.123 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"605c42f8-02a1-11ef-bc93-c7b262605968","created":"2024-04-25T01:17:02.974878+0000","modified":"2024-04-25T01:26:01.175296+0000","last_up_change":"2024-04-25T01:24:21.988539+0000","last_in_change":"2024-04-25T01:24:01.449841+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:20:28.050972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:25:14.398788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:25:15.720822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b40650a-6f71-49c8-b04f-f76ed7b91b72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6803","nonce":1017436960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6805","nonce":1017436960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6809","nonce":1017436960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1017436960},{"type":"v1","addr":"172.21.15.16:6807","nonce":1017436960}]},"public_addr":"172.21.15.16:6803/1017436960","cluster_addr":"172.21.15.16:6805/1017436960","heartbeat_back_addr":"172.21.15.16:6809/1017436960","heartbeat_front_addr":"172.21.15.16:6807/1017436960","state":["exists","up"]},{"osd":1,"uuid":"3d73a4e5-3402-40e8-a625-53872c981f89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6801","nonce":4146278985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6803","nonce":4146278985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6807","nonce":4146278985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4146278985},{"type":"v1","addr":"172.21.15.38:6805","nonce":4146278985}]},"public_addr":"172.21.15.38:6801/4146278985","cluster_addr":"172.21.15.38:6803/4146278985","heartbeat_back_addr":"172.21.15.38:6807/4146278985","heartbeat_front_addr":"172.21.15.38:6805/4146278985","state":["exists","up"]},{"osd":2,"uuid":"ffa52ec5-7d59-482c-bf75-87a3b7f21447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6801","nonce":380124115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6803","nonce":380124115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6807","nonce":380124115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":380124115},{"type":"v1","addr":"172.21.15.156:6805","nonce":380124115}]},"public_addr":"172.21.15.156:6801/380124115","cluster_addr":"172.21.15.156:6803/380124115","heartbeat_back_addr":"172.21.15.156:6807/380124115","heartbeat_front_addr":"172.21.15.156:6805/380124115","state":["exists","up"]},{"osd":3,"uuid":"2c945cb2-a2ff-473b-98a7-2e462b7c59b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6811","nonce":3649873026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6813","nonce":3649873026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6817","nonce":3649873026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3649873026},{"type":"v1","addr":"172.21.15.16:6815","nonce":3649873026}]},"public_addr":"172.21.15.16:6811/3649873026","cluster_addr":"172.21.15.16:6813/3649873026","heartbeat_back_addr":"172.21.15.16:6817/3649873026","heartbeat_front_addr":"172.21.15.16:6815/3649873026","state":["exists","up"]},{"osd":4,"uuid":"5e91cb88-1603-4183-b475-1ab7df79597a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6809","nonce":1109232541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6811","nonce":1109232541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6815","nonce":1109232541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1109232541},{"type":"v1","addr":"172.21.15.38:6813","nonce":1109232541}]},"public_addr":"172.21.15.38:6809/1109232541","cluster_addr":"172.21.15.38:6811/1109232541","heartbeat_back_addr":"172.21.15.38:6815/1109232541","heartbeat_front_addr":"172.21.15.38:6813/1109232541","state":["exists","up"]},{"osd":5,"uuid":"451201e8-0fc6-4319-b7d9-9be4425d6384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6809","nonce":3440527374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6811","nonce":3440527374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6815","nonce":3440527374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3440527374},{"type":"v1","addr":"172.21.15.156:6813","nonce":3440527374}]},"public_addr":"172.21.15.156:6809/3440527374","cluster_addr":"172.21.15.156:6811/3440527374","heartbeat_back_addr":"172.21.15.156:6815/3440527374","heartbeat_front_addr":"172.21.15.156:6813/3440527374","state":["exists","up"]},{"osd":6,"uuid":"f933fcbe-f953-4fb0-8e2f-5eec380563bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6819","nonce":2018477454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6821","nonce":2018477454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6825","nonce":2018477454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2018477454},{"type":"v1","addr":"172.21.15.16:6823","nonce":2018477454}]},"public_addr":"172.21.15.16:6819/2018477454","cluster_addr":"172.21.15.16:6821/2018477454","heartbeat_back_addr":"172.21.15.16:6825/2018477454","heartbeat_front_addr":"172.21.15.16:6823/2018477454","state":["exists","up"]},{"osd":7,"uuid":"da57b8de-edaf-497f-afdb-0d5648bf0c3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6817","nonce":2653364257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6819","nonce":2653364257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6823","nonce":2653364257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2653364257},{"type":"v1","addr":"172.21.15.38:6821","nonce":2653364257}]},"public_addr":"172.21.15.38:6817/2653364257","cluster_addr":"172.21.15.38:6819/2653364257","heartbeat_back_addr":"172.21.15.38:6823/2653364257","heartbeat_front_addr":"172.21.15.38:6821/2653364257","state":["exists","up"]},{"osd":8,"uuid":"d6850fc5-5935-403c-bbe2-c2fc2e73d3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6817","nonce":1052118798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6819","nonce":1052118798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6823","nonce":1052118798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1052118798},{"type":"v1","addr":"172.21.15.156:6821","nonce":1052118798}]},"public_addr":"172.21.15.156:6817/1052118798","cluster_addr":"172.21.15.156:6819/1052118798","heartbeat_back_addr":"172.21.15.156:6823/1052118798","heartbeat_front_addr":"172.21.15.156:6821/1052118798","state":["exists","up"]},{"osd":9,"uuid":"d90517c5-56a1-45f6-adf5-6c2b28702bc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6827","nonce":2042480684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6829","nonce":2042480684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6833","nonce":2042480684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2042480684},{"type":"v1","addr":"172.21.15.16:6831","nonce":2042480684}]},"public_addr":"172.21.15.16:6827/2042480684","cluster_addr":"172.21.15.16:6829/2042480684","heartbeat_back_addr":"172.21.15.16:6833/2042480684","heartbeat_front_addr":"172.21.15.16:6831/2042480684","state":["exists","up"]},{"osd":10,"uuid":"3da7ce51-60f7-45d6-87d3-42baee7a62db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6825","nonce":3034934986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6827","nonce":3034934986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6831","nonce":3034934986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":3034934986},{"type":"v1","addr":"172.21.15.38:6829","nonce":3034934986}]},"public_addr":"172.21.15.38:6825/3034934986","cluster_addr":"172.21.15.38:6827/3034934986","heartbeat_back_addr":"172.21.15.38:6831/3034934986","heartbeat_front_addr":"172.21.15.38:6829/3034934986","state":["exists","up"]},{"osd":11,"uuid":"b8c5bd1d-550a-410c-8247-2dc52f893b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6825","nonce":1283302638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6827","nonce":1283302638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6831","nonce":1283302638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1283302638},{"type":"v1","addr":"172.21.15.156:6829","nonce":1283302638}]},"public_addr":"172.21.15.156:6825/1283302638","cluster_addr":"172.21.15.156:6827/1283302638","heartbeat_back_addr":"172.21.15.156:6831/1283302638","heartbeat_front_addr":"172.21.15.156:6829/1283302638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:19:35.684307+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:00.555923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:25.020880+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:20:50.678984+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:16.882418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:21:41.951399+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:08.622244+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:22:34.556070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:00.374521+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:27.000840+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:23:53.510011+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:19.813012+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/4006719933":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/2073759739":"2024-04-26T01:17:59.526723+0000","172.21.15.16:6801/1190188268":"2024-04-26T01:17:59.526723+0000","172.21.15.16:0/1590801448":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6800/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2428618530":"2024-04-26T01:17:31.392949+0000","172.21.15.16:6801/1662737541":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/2674684320":"2024-04-26T01:17:31.392949+0000","172.21.15.16:0/153993658":"2024-04-26T01:17:59.526723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f100c0a2b90 msgr2=0x7f100c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f100c0a2b90 0x7f100c0a5050 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1024197db0 tx=0x7f1018008040 comp rx=0 tx=0).stop 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 msgr2=0x7f10241968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 0x7f10241968a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f101400c970 tx=0x7f101400ce30 comp rx=0 tx=0).stop 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 shutdown_connections 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f100c0a2b90 0x7f100c0a5050 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1024102250 0x7f102419de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1024101890 0x7f1024196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 --2- 172.21.15.16:0/2784569450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10241006a0 0x7f10241968a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 >> 172.21.15.16:0/2784569450 conn(0x7f10240fbc30 msgr2=0x7f1024104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 shutdown_connections 2024-04-25T01:39:29.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.142+0000 7f102a31b640 1 -- 172.21.15.16:0/2784569450 wait complete. 2024-04-25T01:39:29.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:39:29.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.258+0000 7f7e00977640 1 -- 172.21.15.16:0/3917478091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 msgr2=0x7f7dfc101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.258+0000 7f7e00977640 1 --2- 172.21.15.16:0/3917478091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc101bc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009900 tx=0x7f7de402a410 comp rx=0 tx=0).stop 2024-04-25T01:39:29.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- 172.21.15.16:0/3917478091 shutdown_connections 2024-04-25T01:39:29.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 --2- 172.21.15.16:0/3917478091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dfc103350 0x7f7dfc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 --2- 172.21.15.16:0/3917478091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7dfc102990 0x7f7dfc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 --2- 172.21.15.16:0/3917478091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc101bc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- 172.21.15.16:0/3917478091 >> 172.21.15.16:0/3917478091 conn(0x7f7dfc0fcd10 msgr2=0x7f7dfc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:29.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- 172.21.15.16:0/3917478091 shutdown_connections 2024-04-25T01:39:29.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- 172.21.15.16:0/3917478091 wait complete. 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 Processor -- start 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- start start 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7dfc102990 0x7f7dfc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dfc103350 0x7f7dfc074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc10d2a0 con 0x7f7dfc1017a0 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc10d120 con 0x7f7dfc102990 2024-04-25T01:39:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.259+0000 7f7e00977640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc10d420 con 0x7f7dfc103350 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc0754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51788/0 (socket says 172.21.15.16:51788) 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 -- 172.21.15.16:0/3502176585 learned_addr learned my addr 172.21.15.16:0/3502176585 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7df9d9b640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7dfc102990 0x7f7dfc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfad9d640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dfc103350 0x7f7dfc074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 -- 172.21.15.16:0/3502176585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dfc103350 msgr2=0x7f7dfc074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dfc103350 0x7f7dfc074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 -- 172.21.15.16:0/3502176585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7dfc102990 msgr2=0x7f7dfc073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7dfc102990 0x7f7dfc073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 -- 172.21.15.16:0/3502176585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc074900 con 0x7f7dfc1017a0 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7dfa59c640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc0754f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7de40314e0 tx=0x7f7de40042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de403f030 con 0x7f7dfc1017a0 2024-04-25T01:39:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dfc1a1a10 con 0x7f7dfc1017a0 2024-04-25T01:39:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7de40044d0 con 0x7f7dfc1017a0 2024-04-25T01:39:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4035070 con 0x7f7dfc1017a0 2024-04-25T01:39:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.260+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dfc1a1fb0 con 0x7f7dfc1017a0 2024-04-25T01:39:29.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.261+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dfc108be0 con 0x7f7dfc1017a0 2024-04-25T01:39:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.263+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f7de4042030 con 0x7f7dfc1017a0 2024-04-25T01:39:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.263+0000 7f7df37fe640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7de80a2b90 0x7f7de80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.263+0000 7f7df9d9b640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7de80a2b90 0x7f7de80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.263+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7de40ef990 con 0x7f7dfc1017a0 2024-04-25T01:39:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.264+0000 7f7df9d9b640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7de80a2b90 0x7f7de80a5050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc075260 tx=0x7f7dec00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:29.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.267+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7de40bc870 con 0x7f7dfc1017a0 2024-04-25T01:39:29.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.491+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7dfc062340 con 0x7f7dfc1017a0 2024-04-25T01:39:29.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.493+0000 7f7df37fe640 1 -- 172.21.15.16:0/3502176585 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) ==== 161+0+0 (secure 0 0 0) 0x7f7de40bc170 con 0x7f7dfc1017a0 2024-04-25T01:39:29.476 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:39:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7de80a2b90 msgr2=0x7f7de80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7de80a2b90 0x7f7de80a5050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc075260 tx=0x7f7dec00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 msgr2=0x7f7dfc0754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc0754f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7de40314e0 tx=0x7f7de40042e0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 shutdown_connections 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f7de80a2b90 0x7f7de80a5050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7dfc103350 0x7f7dfc074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7dfc102990 0x7f7dfc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.495+0000 7f7e00977640 1 --2- 172.21.15.16:0/3502176585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dfc1017a0 0x7f7dfc0754f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.496+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 >> 172.21.15.16:0/3502176585 conn(0x7f7dfc0fcd10 msgr2=0x7f7dfc105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.496+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 shutdown_connections 2024-04-25T01:39:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.496+0000 7f7e00977640 1 -- 172.21.15.16:0/3502176585 wait complete. 2024-04-25T01:39:29.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 -- 172.21.15.16:0/2764861458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0103340 msgr2=0x7f13e0109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 --2- 172.21.15.16:0/2764861458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0103340 0x7f13e0109bd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009ca0 tx=0x7f13d802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 -- 172.21.15.16:0/2764861458 shutdown_connections 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 --2- 172.21.15.16:0/2764861458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0103340 0x7f13e0109bd0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 --2- 172.21.15.16:0/2764861458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0102980 0x7f13e0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 --2- 172.21.15.16:0/2764861458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13e0101790 0x7f13e0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 -- 172.21.15.16:0/2764861458 >> 172.21.15.16:0/2764861458 conn(0x7f13e00fcd20 msgr2=0x7f13e00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 -- 172.21.15.16:0/2764861458 shutdown_connections 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.614+0000 7f13e7638640 1 -- 172.21.15.16:0/2764861458 wait complete. 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 Processor -- start 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 -- start start 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0101790 0x7f13e0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 0x7f13e0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13e0103340 0x7f13e019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13e010d2f0 con 0x7f13e0102980 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f13e010d170 con 0x7f13e0103340 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e7638640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f13e010d470 con 0x7f13e0101790 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 0x7f13e0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 0x7f13e0197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51808/0 (socket says 172.21.15.16:51808) 2024-04-25T01:39:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 -- 172.21.15.16:0/2712459539 learned_addr learned my addr 172.21.15.16:0/2712459539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e5bd5640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13e0103340 0x7f13e019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e53d4640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0101790 0x7f13e0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 -- 172.21.15.16:0/2712459539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0101790 msgr2=0x7f13e0197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0101790 0x7f13e0197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 -- 172.21.15.16:0/2712459539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13e0103340 msgr2=0x7f13e019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13e0103340 0x7f13e019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 -- 172.21.15.16:0/2712459539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e019f5b0 con 0x7f13e0102980 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.615+0000 7f13e4bd3640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 0x7f13e0197ed0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f13d000e9f0 tx=0x7f13d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.616+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d000cd80 con 0x7f13e0102980 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.616+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13d00108e0 con 0x7f13e0102980 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.616+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e019f8a0 con 0x7f13e0102980 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.616+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0018920 con 0x7f13e0102980 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.616+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e019fdf0 con 0x7f13e0102980 2024-04-25T01:39:29.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.617+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e0108ab0 con 0x7f13e0102980 2024-04-25T01:39:29.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.618+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f13d0010a90 con 0x7f13e0102980 2024-04-25T01:39:29.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.618+0000 7f13d67fc640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f13cc0a2b90 0x7f13cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.618+0000 7f13e53d4640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f13cc0a2b90 0x7f13cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.619+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f13d00cf540 con 0x7f13e0102980 2024-04-25T01:39:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.619+0000 7f13e53d4640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f13cc0a2b90 0x7f13cc0a5050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f13dc005fd0 tx=0x7f13dc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.623+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f13d009c320 con 0x7f13e0102980 2024-04-25T01:39:29.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:29.849+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f13e0062340 con 0x7f13e0102980 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a[29148]: 2024-04-25T01:39:29.491+0000 7f7354257700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: pgmap v721: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/1156200151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2784569450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3502176585' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/3502176585' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:39:29.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:29 smithi016 ceph-mon[29153]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T01:39:29.969 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: ignoring --setuser ceph since I am not root 2024-04-25T01:39:29.969 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: ignoring --setgroup ceph since I am not root 2024-04-25T01:39:29.969 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mds[62871]: -- 172.21.15.38:0/2972265196 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5588340f3a40 con 0x5588340c6c00 2024-04-25T01:39:29.969 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: starting mds.b at 2024-04-25T01:39:29.970 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e[62242]: ignoring --setuser ceph since I am not root 2024-04-25T01:39:29.970 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e[62242]: ignoring --setgroup ceph since I am not root 2024-04-25T01:39:29.970 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e[62242]: starting mds.e at 2024-04-25T01:39:29.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: pgmap v721: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-25T01:39:29.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/1156200151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2784569450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3502176585' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/3502176585' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mon[32841]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h[62555]: ignoring --setuser ceph since I am not root 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h[62555]: ignoring --setgroup ceph since I am not root 2024-04-25T01:39:29.971 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-mds[62559]: -- 172.21.15.38:0/4156289003 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607938e9a40 con 0x5607938bcc00 2024-04-25T01:39:29.972 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:39:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h[62555]: starting mds.h at 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f[63314]: ignoring --setuser ceph since I am not root 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f[63314]: ignoring --setgroup ceph since I am not root 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mds[63318]: -- 172.21.15.156:0/1633508400 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5570f20cda40 con 0x5570f20a0c00 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f[63314]: starting mds.f at 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i[63930]: ignoring --setuser ceph since I am not root 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i[63930]: ignoring --setgroup ceph since I am not root 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mds[63934]: -- 172.21.15.156:0/829659342 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563908dc3a40 con 0x563908d96c00 2024-04-25T01:39:30.004 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i[63930]: starting mds.i at 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: pgmap v721: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/1156200151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2784569450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3502176585' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/3502176585' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:39:30.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mon[32631]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T01:39:30.006 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c[63622]: ignoring --setuser ceph since I am not root 2024-04-25T01:39:30.006 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c[63622]: ignoring --setgroup ceph since I am not root 2024-04-25T01:39:30.006 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-mds[63626]: -- 172.21.15.156:0/3747903458 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557685559a40 con 0x55768552d000 2024-04-25T01:39:30.006 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:39:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c[63622]: starting mds.c at 2024-04-25T01:39:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.495+0000 7f13d67fc640 1 -- 172.21.15.16:0/2712459539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) ==== 104+0+0 (secure 0 0 0) 0x7f13d009bc20 con 0x7f13e0102980 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f13cc0a2b90 msgr2=0x7f13cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f13cc0a2b90 0x7f13cc0a5050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f13dc005fd0 tx=0x7f13dc005850 comp rx=0 tx=0).stop 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 msgr2=0x7f13e0197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 0x7f13e0197ed0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f13d000e9f0 tx=0x7f13d000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 shutdown_connections 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f13cc0a2b90 0x7f13cc0a5050 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13e0103340 0x7f13e019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0102980 0x7f13e0197ed0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 --2- 172.21.15.16:0/2712459539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0101790 0x7f13e0197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 >> 172.21.15.16:0/2712459539 conn(0x7f13e00fcd20 msgr2=0x7f13e0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 shutdown_connections 2024-04-25T01:39:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.498+0000 7f13e7638640 1 -- 172.21.15.16:0/2712459539 wait complete. 2024-04-25T01:39:30.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:39:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 -- 172.21.15.16:0/2963271738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 msgr2=0x7fc5d4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/2963271738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4101cf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009900 tx=0x7fc5c402a410 comp rx=0 tx=0).stop 2024-04-25T01:39:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 -- 172.21.15.16:0/2963271738 shutdown_connections 2024-04-25T01:39:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/2963271738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5d4102230 0x7fc5d4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/2963271738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4101cf0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/2963271738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5d4100680 0x7fc5d4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 -- 172.21.15.16:0/2963271738 >> 172.21.15.16:0/2963271738 conn(0x7fc5d40fbbf0 msgr2=0x7fc5d40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 -- 172.21.15.16:0/2963271738 shutdown_connections 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.628+0000 7fc5dbee1640 1 -- 172.21.15.16:0/2963271738 wait complete. 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 Processor -- start 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 -- start start 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5d4100680 0x7fc5d4104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5d4102230 0x7fc5d4103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d410c190 con 0x7fc5d4101870 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d410c010 con 0x7fc5d4100680 2024-04-25T01:39:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5dbee1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d410c310 con 0x7fc5d4102230 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5d947c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5d947c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51834/0 (socket says 172.21.15.16:51834) 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5d947c640 1 -- 172.21.15.16:0/115295895 learned_addr learned my addr 172.21.15.16:0/115295895 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5d9c7d640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5d4100680 0x7fc5d4104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.629+0000 7fc5da47e640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5d4102230 0x7fc5d4103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5d947c640 1 -- 172.21.15.16:0/115295895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5d4102230 msgr2=0x7fc5d4103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5d947c640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5d4102230 0x7fc5d4103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5d947c640 1 -- 172.21.15.16:0/115295895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5d4100680 msgr2=0x7fc5d4104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5d947c640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5d4100680 0x7fc5d4104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5d947c640 1 -- 172.21.15.16:0/115295895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d4103a20 con 0x7fc5d4101870 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5d947c640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4102c50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005940 tx=0x7fc5c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c403f030 con 0x7fc5d4101870 2024-04-25T01:39:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5c4003cb0 con 0x7fc5d4101870 2024-04-25T01:39:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d41a1aa0 con 0x7fc5d4101870 2024-04-25T01:39:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4035070 con 0x7fc5d4101870 2024-04-25T01:39:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.630+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d41a1ff0 con 0x7fc5d4101870 2024-04-25T01:39:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.631+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d41a21c0 con 0x7fc5d4101870 2024-04-25T01:39:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.633+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fc5c4042030 con 0x7fc5d4101870 2024-04-25T01:39:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.633+0000 7fc5caffd640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc5c00a2c10 0x7fc5c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.633+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fc5c40efd50 con 0x7fc5d4101870 2024-04-25T01:39:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.633+0000 7fc5d9c7d640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc5c00a2c10 0x7fc5c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.634+0000 7fc5d9c7d640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc5c00a2c10 0x7fc5c00a50d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0005fd0 tx=0x7fc5d000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.637+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc5c40bc4e0 con 0x7fc5d4101870 2024-04-25T01:39:30.783 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2712459539' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: Cluster is now healthy 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2712459539' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6834/1441290559,v1:172.21.15.156:6835/1441290559] up:boot 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6834/3482536148,v1:172.21.15.38:6837/3482536148] up:boot 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6836/4016907599,v1:172.21.15.38:6839/4016907599] up:boot 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.38:6835/815844530,v1:172.21.15.38:6838/815844530] up:boot 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6836/717774921,v1:172.21.15.156:6838/717774921] up:boot 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: mds.? [v2:172.21.15.156:6837/1653773910,v1:172.21.15.156:6839/1653773910] up:boot 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: fsmap 12 up:standby 2024-04-25T01:39:30.784 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:39:30.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:39:30.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:39:30.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:39:30.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:39:30.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:30 smithi016 ceph-mon[29153]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:39:30.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:30.823+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc5d41a28c0 con 0x7fc5d4101870 2024-04-25T01:39:30.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2712459539' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:39:30.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:39:30.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: Cluster is now healthy 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2712459539' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6834/1441290559,v1:172.21.15.156:6835/1441290559] up:boot 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6834/3482536148,v1:172.21.15.38:6837/3482536148] up:boot 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6836/4016907599,v1:172.21.15.38:6839/4016907599] up:boot 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.38:6835/815844530,v1:172.21.15.38:6838/815844530] up:boot 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6836/717774921,v1:172.21.15.156:6838/717774921] up:boot 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: mds.? [v2:172.21.15.156:6837/1653773910,v1:172.21.15.156:6839/1653773910] up:boot 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: fsmap 12 up:standby 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:39:30.970 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:39:30.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:39:30.971 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:30 smithi038 ceph-mon[32841]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2712459539' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: Cluster is now healthy 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2712459539' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6834/1441290559,v1:172.21.15.156:6835/1441290559] up:boot 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6834/3482536148,v1:172.21.15.38:6837/3482536148] up:boot 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6836/4016907599,v1:172.21.15.38:6839/4016907599] up:boot 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.38:6835/815844530,v1:172.21.15.38:6838/815844530] up:boot 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6836/717774921,v1:172.21.15.156:6838/717774921] up:boot 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: mds.? [v2:172.21.15.156:6837/1653773910,v1:172.21.15.156:6839/1653773910] up:boot 2024-04-25T01:39:31.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: fsmap 12 up:standby 2024-04-25T01:39:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:39:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:39:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:39:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:39:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:39:31.005 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:30 smithi156 ceph-mon[32631]: from='mgr.14152 172.21.15.16:0/2251504136' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:39:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.673+0000 7fc5caffd640 1 -- 172.21.15.16:0/115295895 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) ==== 181+0+0 (secure 0 0 0) 0x7fc5c40c07e0 con 0x7fc5d4101870 2024-04-25T01:39:31.656 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:39:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc5c00a2c10 msgr2=0x7fc5c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc5c00a2c10 0x7fc5c00a50d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0005fd0 tx=0x7fc5d000b410 comp rx=0 tx=0).stop 2024-04-25T01:39:31.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 msgr2=0x7fc5d4102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4102c50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005940 tx=0x7fc5c40042f0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 shutdown_connections 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7fc5c00a2c10 0x7fc5c00a50d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5d4102230 0x7fc5d4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5d4101870 0x7fc5d4102c50 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 --2- 172.21.15.16:0/115295895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5d4100680 0x7fc5d4104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.676+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 >> 172.21.15.16:0/115295895 conn(0x7fc5d40fbbf0 msgr2=0x7fc5d40688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.677+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 shutdown_connections 2024-04-25T01:39:31.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.677+0000 7fc5dbee1640 1 -- 172.21.15.16:0/115295895 wait complete. 2024-04-25T01:39:31.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:39:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 -- 172.21.15.16:0/1949228357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d8102260 msgr2=0x7f02d8104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 --2- 172.21.15.16:0/1949228357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d8102260 0x7f02d8104650 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009900 tx=0x7f02c002a410 comp rx=0 tx=0).stop 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 -- 172.21.15.16:0/1949228357 shutdown_connections 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 --2- 172.21.15.16:0/1949228357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02d81074c0 0x7f02d81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 --2- 172.21.15.16:0/1949228357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d8104b90 0x7f02d8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 --2- 172.21.15.16:0/1949228357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d8102260 0x7f02d8104650 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 -- 172.21.15.16:0/1949228357 >> 172.21.15.16:0/1949228357 conn(0x7f02d80fbbf0 msgr2=0x7f02d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 -- 172.21.15.16:0/1949228357 shutdown_connections 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 -- 172.21.15.16:0/1949228357 wait complete. 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 Processor -- start 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.809+0000 7f02de288640 1 -- start start 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02de288640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d8102260 0x7f02d8101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02de288640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02d8104b90 0x7f02d8100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:31.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02de288640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 0x7f02d8100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02de288640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02d810d2d0 con 0x7f02d81074c0 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02de288640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02d810d150 con 0x7f02d8104b90 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02de288640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02d810d450 con 0x7f02d8102260 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 0x7f02d8100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 0x7f02d8100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46290/0 (socket says 172.21.15.16:46290) 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 -- 172.21.15.16:0/2505312482 learned_addr learned my addr 172.21.15.16:0/2505312482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02d7fff640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d8102260 0x7f02d8101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02d77fe640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02d8104b90 0x7f02d8100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 -- 172.21.15.16:0/2505312482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d8102260 msgr2=0x7f02d8101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d8102260 0x7f02d8101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 -- 172.21.15.16:0/2505312482 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02d8104b90 msgr2=0x7f02d8100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02d8104b90 0x7f02d8100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 -- 172.21.15.16:0/2505312482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d8100fe0 con 0x7f02d81074c0 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.810+0000 7f02dc825640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 0x7f02d8100750 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009fd0 tx=0x7f02d000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.811+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0017070 con 0x7f02d81074c0 2024-04-25T01:39:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.811+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d81a1ba0 con 0x7f02d81074c0 2024-04-25T01:39:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.811+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02d0003e20 con 0x7f02d81074c0 2024-04-25T01:39:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.811+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d81a20a0 con 0x7f02d81074c0 2024-04-25T01:39:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.811+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0020060 con 0x7f02d81074c0 2024-04-25T01:39:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.812+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d80fd6e0 con 0x7f02d81074c0 2024-04-25T01:39:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.813+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f02d0003970 con 0x7f02d81074c0 2024-04-25T01:39:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.814+0000 7f02d57fa640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02c40a2b90 0x7f02c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.814+0000 7f02d7fff640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02c40a2b90 0x7f02c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.814+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8002+0+0 (secure 0 0 0) 0x7f02d00cf0b0 con 0x7f02d81074c0 2024-04-25T01:39:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.814+0000 7f02d7fff640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02c40a2b90 0x7f02c40a5050 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02c00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:31.819+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02d009c0a0 con 0x7f02d81074c0 2024-04-25T01:39:31.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:31 smithi038 ceph-mon[32841]: pgmap v723: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 7 op/s 2024-04-25T01:39:31.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:31 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/115295895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:39:31.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.003+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f02d802d170 con 0x7f02d81074c0 2024-04-25T01:39:32.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:31 smithi156 ceph-mon[32631]: pgmap v723: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 7 op/s 2024-04-25T01:39:32.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:31 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/115295895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:39:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:31 smithi016 ceph-mon[29153]: pgmap v723: 97 pgs: 97 active+clean; 1.3 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 7 op/s 2024-04-25T01:39:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:31 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/115295895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:39:32.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.679+0000 7f02d57fa640 1 -- 172.21.15.16:0/2505312482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) ==== 169+0+0 (secure 0 0 0) 0x7f02d009b9a0 con 0x7f02d81074c0 2024-04-25T01:39:32.662 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-04-25T01:39:32.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.683+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02c40a2b90 msgr2=0x7f02c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:32.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.683+0000 7f02de288640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02c40a2b90 0x7f02c40a5050 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02c00314a0 comp rx=0 tx=0).stop 2024-04-25T01:39:32.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.683+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 msgr2=0x7f02d8100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:32.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.683+0000 7f02de288640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 0x7f02d8100750 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009fd0 tx=0x7f02d000b260 comp rx=0 tx=0).stop 2024-04-25T01:39:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 shutdown_connections 2024-04-25T01:39:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:6800/1051975299,v1:172.21.15.16:6801/1051975299] conn(0x7f02c40a2b90 0x7f02c40a5050 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:32.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d81074c0 0x7f02d8100750 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02d8104b90 0x7f02d8100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 --2- 172.21.15.16:0/2505312482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d8102260 0x7f02d8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 >> 172.21.15.16:0/2505312482 conn(0x7f02d80fbbf0 msgr2=0x7f02d8102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 shutdown_connections 2024-04-25T01:39:32.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-25T01:39:32.685+0000 7f02de288640 1 -- 172.21.15.16:0/2505312482 wait complete. 2024-04-25T01:39:32.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:39:32.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:39:32.763 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:39:32.790 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T01:39:32.791 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:39:32.820 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:39:32.839 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:39:32.853 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T01:39:32.854 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T01:39:32.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:39:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:32 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/115295895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:39:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:32 smithi016 ceph-mon[29153]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:39:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:32 smithi016 ceph-mon[29153]: from='client.? 172.21.15.16:0/2505312482' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:39:32.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:32 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/115295895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:39:32.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:32 smithi038 ceph-mon[32841]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:39:32.969 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:32 smithi038 ceph-mon[32841]: from='client.? 172.21.15.16:0/2505312482' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:39:33.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:32 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/115295895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:39:33.003 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:32 smithi156 ceph-mon[32631]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:39:33.004 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:32 smithi156 ceph-mon[32631]: from='client.? 172.21.15.16:0/2505312482' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:39:33.203 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:32 smithi016 systemd[1]: Stopping Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:33.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a[29148]: 2024-04-25T01:39:33.220+0000 7f7359a62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:33.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a[29148]: 2024-04-25T01:39:33.220+0000 7f7359a62700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T01:39:33.545 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:33 smithi016 podman[94137]: 2024-04-25 01:39:33.252362956 +0000 UTC m=+0.164048840 container died 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T01:39:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:33 smithi016 podman[94137]: 2024-04-25 01:39:33.67989224 +0000 UTC m=+0.591578115 container cleanup 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9) 2024-04-25T01:39:33.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:33 smithi016 bash[94137]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:39:34.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T01:39:34.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:33 smithi016 podman[94151]: 2024-04-25 01:39:33.923553437 +0000 UTC m=+0.663261433 container remove 80597ad1e310667255e7adfb08055b1a7807501c8975b752e9bd738aee572afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9) 2024-04-25T01:39:34.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:34 smithi016 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:39:34.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:34 smithi016 systemd[1]: Stopped Ceph mon.a for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:39:34.173 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 25 01:39:34 smithi016 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.a.service: Consumed 21.922s CPU time. 2024-04-25T01:39:34.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:34.189 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T01:39:34.189 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T01:39:34.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.b 2024-04-25T01:39:34.469 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:34 smithi038 systemd[1]: Stopping Ceph mon.b for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:34.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b[32837]: 2024-04-25T01:39:34.526+0000 7fd76cd64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:34.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b[32837]: 2024-04-25T01:39:34.526+0000 7fd76cd64700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T01:39:34.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:34 smithi038 podman[67328]: 2024-04-25 01:39:34.65002845 +0000 UTC m=+0.245522773 container died 97c40aa9d4810b99251505da6d09efd8a3c61619c42455529904d9f9931614be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T01:39:35.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:35 smithi038 podman[67328]: 2024-04-25 01:39:35.100711139 +0000 UTC m=+0.696205459 container cleanup 97c40aa9d4810b99251505da6d09efd8a3c61619c42455529904d9f9931614be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T01:39:35.219 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 25 01:39:35 smithi038 bash[67328]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-b 2024-04-25T01:39:35.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T01:39:35.672 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:35.672 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T01:39:35.672 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T01:39:35.673 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.c 2024-04-25T01:39:35.993 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:35 smithi156 systemd[1]: Stopping Ceph mon.c for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:36.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-c[32614]: 2024-04-25T01:39:36.009+0000 7fb8abdd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:36.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-c[32614]: 2024-04-25T01:39:36.009+0000 7fb8abdd6700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T01:39:36.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:36 smithi156 podman[68471]: 2024-04-25 01:39:36.245376743 +0000 UTC m=+0.356488499 container died d84995c16b88fdbfac3b4663819ce861223d4cb1a281bc9cbf58be1d064f649f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T01:39:36.937 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:36 smithi156 podman[68471]: 2024-04-25 01:39:36.619699607 +0000 UTC m=+0.730811365 container cleanup d84995c16b88fdbfac3b4663819ce861223d4cb1a281bc9cbf58be1d064f649f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T01:39:36.937 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 01:39:36 smithi156 bash[68471]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mon-c 2024-04-25T01:39:37.075 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T01:39:37.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:37.123 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T01:39:37.123 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T01:39:37.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:39:37.451 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:39:37 smithi016 systemd[1]: Stopping Ceph mgr.x for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:37.752 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:39:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x[29386]: 2024-04-25T01:39:37.467+0000 7f038fd61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:37.752 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:39:37 smithi016 podman[94256]: 2024-04-25 01:39:37.49939187 +0000 UTC m=+0.161192587 container died 4c4a282d44c92ab066678b2930b317866cb10b19541c154827f4c68f6e9eee8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T01:39:38.113 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:39:37 smithi016 podman[94256]: 2024-04-25 01:39:37.877620165 +0000 UTC m=+0.539420888 container cleanup 4c4a282d44c92ab066678b2930b317866cb10b19541c154827f4c68f6e9eee8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T01:39:38.113 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 25 01:39:37 smithi016 bash[94256]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-x 2024-04-25T01:39:38.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T01:39:38.330 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:38.331 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T01:39:38.331 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T01:39:38.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.y 2024-04-25T01:39:38.677 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:39:38 smithi038 systemd[1]: Stopping Ceph mgr.y for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:38.966 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:39:38 smithi038 podman[67450]: 2024-04-25 01:39:38.695472274 +0000 UTC m=+0.142830833 container died 68bae53df6defa01d510382883862228a561c0f89e49aac85cd4308252241afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:39:39.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:39:39 smithi038 podman[67450]: 2024-04-25 01:39:39.067019657 +0000 UTC m=+0.514378230 container cleanup 68bae53df6defa01d510382883862228a561c0f89e49aac85cd4308252241afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T01:39:39.219 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 25 01:39:39 smithi038 bash[67450]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-y 2024-04-25T01:39:39.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T01:39:39.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:39.517 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T01:39:39.517 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T01:39:39.517 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.z 2024-04-25T01:39:39.853 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:39:39 smithi156 systemd[1]: Stopping Ceph mgr.z for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:40.133 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:39:39 smithi156 podman[68593]: 2024-04-25 01:39:39.871035406 +0000 UTC m=+0.146862753 container died ff88161473000fcaee15ca0aab27961d8b2d23b3b6394015336b60fe622b3644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:40.485 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:39:40 smithi156 podman[68593]: 2024-04-25 01:39:40.251344987 +0000 UTC m=+0.527172338 container cleanup ff88161473000fcaee15ca0aab27961d8b2d23b3b6394015336b60fe622b3644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T01:39:40.485 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 01:39:40 smithi156 bash[68593]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mgr-z 2024-04-25T01:39:40.658 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T01:39:40.691 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:40.692 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T01:39:40.692 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T01:39:40.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.0 2024-04-25T01:39:41.024 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:40 smithi016 systemd[1]: Stopping Ceph osd.0 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:41.363 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0[43621]: 2024-04-25T01:39:41.041+0000 7fcbb48bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:41.363 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0[43621]: 2024-04-25T01:39:41.041+0000 7fcbb48bb700 -1 osd.0 90 *** Got signal Terminated *** 2024-04-25T01:39:41.363 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0[43621]: 2024-04-25T01:39:41.041+0000 7fcbb48bb700 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:46.710 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:46 smithi016 podman[94377]: 2024-04-25 01:39:46.440860128 +0000 UTC m=+5.529329873 container died db916fc9e14e09bb007531706a70637e85cce12335dd7e7c7a5e5f650ff0ffc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:39:47.109 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:46 smithi016 podman[94377]: 2024-04-25 01:39:46.816991496 +0000 UTC m=+5.905461257 container cleanup db916fc9e14e09bb007531706a70637e85cce12335dd7e7c7a5e5f650ff0ffc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T01:39:47.109 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:46 smithi016 bash[94377]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0 2024-04-25T01:39:47.363 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:47 smithi016 podman[94394]: 2024-04-25 01:39:47.127001366 +0000 UTC m=+0.686025906 container remove db916fc9e14e09bb007531706a70637e85cce12335dd7e7c7a5e5f650ff0ffc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T01:39:47.662 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:47 smithi016 podman[94487]: 2024-04-25 01:39:47.332898617 +0000 UTC m=+0.018617700 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:47.662 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:47 smithi016 podman[94487]: 2024-04-25 01:39:47.494821605 +0000 UTC m=+0.180540709 container create 6bf7d53ff278bf6fe9ea51b091453e32c5005384bbd622252e0a97f487eeb83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:39:48.113 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:47 smithi016 podman[94487]: 2024-04-25 01:39:47.806648109 +0000 UTC m=+0.492367173 container init 6bf7d53ff278bf6fe9ea51b091453e32c5005384bbd622252e0a97f487eeb83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T01:39:48.114 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:47 smithi016 podman[94487]: 2024-04-25 01:39:47.810051207 +0000 UTC m=+0.495770267 container start 6bf7d53ff278bf6fe9ea51b091453e32c5005384bbd622252e0a97f487eeb83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T01:39:48.114 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:47 smithi016 podman[94487]: 2024-04-25 01:39:47.914912866 +0000 UTC m=+0.600631938 container attach 6bf7d53ff278bf6fe9ea51b091453e32c5005384bbd622252e0a97f487eeb83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T01:39:48.685 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 25 01:39:48 smithi016 podman[94487]: 2024-04-25 01:39:48.389097853 +0000 UTC m=+1.074816918 container died 6bf7d53ff278bf6fe9ea51b091453e32c5005384bbd622252e0a97f487eeb83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-25T01:39:49.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T01:39:49.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:49.068 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T01:39:49.068 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T01:39:49.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.1 2024-04-25T01:39:49.432 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:49 smithi038 systemd[1]: Stopping Ceph osd.1 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:49.720 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1[37410]: 2024-04-25T01:39:49.448+0000 7f06db2b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:49.720 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1[37410]: 2024-04-25T01:39:49.448+0000 7f06db2b6700 -1 osd.1 90 *** Got signal Terminated *** 2024-04-25T01:39:49.720 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1[37410]: 2024-04-25T01:39:49.448+0000 7f06db2b6700 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:39:55.220 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:54 smithi038 podman[67589]: 2024-04-25 01:39:54.984526626 +0000 UTC m=+5.712772514 container died 5e6891697200351046f3d4f4754bcae09847efc860729d1699d2871757444d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:39:55.364 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:55 smithi038 podman[67589]: 2024-04-25 01:39:55.380478313 +0000 UTC m=+6.108724206 container cleanup 5e6891697200351046f3d4f4754bcae09847efc860729d1699d2871757444d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T01:39:55.581 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:55 smithi038 bash[67589]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1 2024-04-25T01:39:55.970 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:55 smithi038 podman[67608]: 2024-04-25 01:39:55.598936845 +0000 UTC m=+0.614341830 container remove 5e6891697200351046f3d4f4754bcae09847efc860729d1699d2871757444d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:39:56.361 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:56 smithi038 podman[67704]: 2024-04-25 01:39:55.965711146 +0000 UTC m=+0.018747880 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:39:56.361 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:56 smithi038 podman[67704]: 2024-04-25 01:39:56.083288571 +0000 UTC m=+0.136325327 container create 6b5c50b6d93cb31c9e3757c06cb1b4335ed3d3dc3bcdf361779fc9ae2ac6e582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:39:56.361 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:56 smithi038 podman[67704]: 2024-04-25 01:39:56.286100287 +0000 UTC m=+0.339137001 container init 6b5c50b6d93cb31c9e3757c06cb1b4335ed3d3dc3bcdf361779fc9ae2ac6e582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T01:39:56.361 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:56 smithi038 podman[67704]: 2024-04-25 01:39:56.289202803 +0000 UTC m=+0.342239518 container start 6b5c50b6d93cb31c9e3757c06cb1b4335ed3d3dc3bcdf361779fc9ae2ac6e582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:56.720 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:56 smithi038 podman[67704]: 2024-04-25 01:39:56.377753735 +0000 UTC m=+0.430790476 container attach 6b5c50b6d93cb31c9e3757c06cb1b4335ed3d3dc3bcdf361779fc9ae2ac6e582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T01:39:56.720 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 25 01:39:56 smithi038 podman[67704]: 2024-04-25 01:39:56.596614886 +0000 UTC m=+0.649651601 container died 6b5c50b6d93cb31c9e3757c06cb1b4335ed3d3dc3bcdf361779fc9ae2ac6e582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T01:39:57.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T01:39:57.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:39:57.287 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T01:39:57.287 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T01:39:57.287 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.2 2024-04-25T01:39:57.606 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:39:57 smithi156 systemd[1]: Stopping Ceph osd.2 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:39:58.004 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:39:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2[37788]: 2024-04-25T01:39:57.622+0000 7fd4ed2b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:39:58.004 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:39:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2[37788]: 2024-04-25T01:39:57.622+0000 7fd4ed2b2700 -1 osd.2 90 *** Got signal Terminated *** 2024-04-25T01:39:58.005 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:39:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2[37788]: 2024-04-25T01:39:57.622+0000 7fd4ed2b2700 -1 osd.2 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:03.339 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:03 smithi156 podman[68729]: 2024-04-25 01:40:03.040494159 +0000 UTC m=+5.540885561 container died 15408ca9b32a31aaa587ec188c53f7331c89e3f0a41583bcfc2eeee9666892db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:40:03.674 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:03 smithi156 podman[68729]: 2024-04-25 01:40:03.456025129 +0000 UTC m=+5.956416555 container cleanup 15408ca9b32a31aaa587ec188c53f7331c89e3f0a41583bcfc2eeee9666892db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:03.674 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:03 smithi156 bash[68729]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2 2024-04-25T01:40:04.004 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:03 smithi156 podman[68749]: 2024-04-25 01:40:03.690675727 +0000 UTC m=+0.652678811 container remove 15408ca9b32a31aaa587ec188c53f7331c89e3f0a41583bcfc2eeee9666892db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T01:40:04.410 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:04 smithi156 podman[68844]: 2024-04-25 01:40:04.032133797 +0000 UTC m=+0.017720880 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:04.410 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:04 smithi156 podman[68844]: 2024-04-25 01:40:04.166616441 +0000 UTC m=+0.152203524 container create 6b8a3cba80848a2095e9da22e14136772f8a50b0be83b40b190b4efc6838031b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T01:40:04.411 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:04 smithi156 podman[68844]: 2024-04-25 01:40:04.42752832 +0000 UTC m=+0.413115403 container init 6b8a3cba80848a2095e9da22e14136772f8a50b0be83b40b190b4efc6838031b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:40:04.703 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:04 smithi156 podman[68844]: 2024-04-25 01:40:04.430707096 +0000 UTC m=+0.416294184 container start 6b8a3cba80848a2095e9da22e14136772f8a50b0be83b40b190b4efc6838031b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T01:40:04.703 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:04 smithi156 podman[68844]: 2024-04-25 01:40:04.527522554 +0000 UTC m=+0.513109642 container attach 6b8a3cba80848a2095e9da22e14136772f8a50b0be83b40b190b4efc6838031b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T01:40:05.005 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 01:40:04 smithi156 podman[68844]: 2024-04-25 01:40:04.719670728 +0000 UTC m=+0.705257810 container died 6b8a3cba80848a2095e9da22e14136772f8a50b0be83b40b190b4efc6838031b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T01:40:05.411 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T01:40:05.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:05.451 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T01:40:05.451 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T01:40:05.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.3 2024-04-25T01:40:05.781 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:05 smithi016 systemd[1]: Stopping Ceph osd.3 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:06.114 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:05 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:05.797+0000 7f8660b5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:06.114 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:05 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:05.797+0000 7f8660b5c700 -1 osd.3 90 *** Got signal Terminated *** 2024-04-25T01:40:06.114 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:05 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:05.797+0000 7f8660b5c700 -1 osd.3 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:07.864 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:07 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:07.530+0000 7f865d150700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.307502+0000 front 2024-04-25T01:39:43.307446+0000 (oldest deadline 2024-04-25T01:40:06.807625+0000) 2024-04-25T01:40:08.864 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:08 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:08.482+0000 7f865d150700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.307502+0000 front 2024-04-25T01:39:43.307446+0000 (oldest deadline 2024-04-25T01:40:06.807625+0000) 2024-04-25T01:40:08.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:08 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:08.496+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:09.864 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:09 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:09.476+0000 7f865d150700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.307502+0000 front 2024-04-25T01:39:43.307446+0000 (oldest deadline 2024-04-25T01:40:06.807625+0000) 2024-04-25T01:40:09.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:09 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:09.507+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:10.364 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:10 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:10.047+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:10.371 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:10.040+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:10.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:10.387+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:10.864 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:10 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3[50169]: 2024-04-25T01:40:10.508+0000 7f865d150700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.307502+0000 front 2024-04-25T01:39:43.307446+0000 (oldest deadline 2024-04-25T01:40:06.807625+0000) 2024-04-25T01:40:10.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:10 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:10.469+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:10.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:10 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:10.586+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:11.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:10.999+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:11.364 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:11 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:11.096+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:11.365 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:11 smithi016 podman[94596]: 2024-04-25 01:40:11.294725202 +0000 UTC m=+5.624376113 container died 58477582aa77b71c8c2e0e0e51ff2b0bd60e1c5efd48ec93840a0e2876001802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:40:11.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:11.351+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:11.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:11 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:11.453+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:11.865 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:11 smithi016 podman[94596]: 2024-04-25 01:40:11.673279239 +0000 UTC m=+6.002930157 container cleanup 58477582aa77b71c8c2e0e0e51ff2b0bd60e1c5efd48ec93840a0e2876001802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T01:40:11.865 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:11 smithi016 bash[94596]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3 2024-04-25T01:40:11.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:11 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:11.614+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:12.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:11.970+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:12.258 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:12 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:12.047+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:12.259 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:11 smithi016 podman[94625]: 2024-04-25 01:40:11.941394302 +0000 UTC m=+0.648756234 container remove 58477582aa77b71c8c2e0e0e51ff2b0bd60e1c5efd48ec93840a0e2876001802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:12.537 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:12 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:12.444+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:12.537 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:12 smithi016 podman[94716]: 2024-04-25 01:40:12.275705702 +0000 UTC m=+0.093564938 container create 23cf42b13f3da410cfe9f933f171a9a5acc28bc06140d27ff66e4eb016509126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:40:12.537 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:12 smithi016 podman[94716]: 2024-04-25 01:40:12.200752825 +0000 UTC m=+0.018612025 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:12.537 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:12 smithi016 podman[94716]: 2024-04-25 01:40:12.553756322 +0000 UTC m=+0.371615531 container init 23cf42b13f3da410cfe9f933f171a9a5acc28bc06140d27ff66e4eb016509126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T01:40:12.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:12.393+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:12.843 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:12 smithi016 podman[94716]: 2024-04-25 01:40:12.557508263 +0000 UTC m=+0.375367461 container start 23cf42b13f3da410cfe9f933f171a9a5acc28bc06140d27ff66e4eb016509126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T01:40:12.843 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:12 smithi016 podman[94716]: 2024-04-25 01:40:12.662075089 +0000 UTC m=+0.479934286 container attach 23cf42b13f3da410cfe9f933f171a9a5acc28bc06140d27ff66e4eb016509126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T01:40:12.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:12 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:12.606+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:13.114 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:13 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:13.090+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:13.114 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 25 01:40:12 smithi016 podman[94716]: 2024-04-25 01:40:12.859357857 +0000 UTC m=+0.677217047 container died 23cf42b13f3da410cfe9f933f171a9a5acc28bc06140d27ff66e4eb016509126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T01:40:13.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:12.953+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:13.614 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:13 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:13.399+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:13.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T01:40:13.731 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:13.731 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T01:40:13.731 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T01:40:13.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.4 2024-04-25T01:40:13.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:13.372+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:13.971 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:13 smithi038 systemd[1]: Stopping Ceph osd.4 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:13.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:13 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:13.562+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:14.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:13.954+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:14.364 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:14 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:14.081+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:14.471 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:40:14.076+0000 7f26abc50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:14.471 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:40:14.076+0000 7f26abc50700 -1 osd.4 90 *** Got signal Terminated *** 2024-04-25T01:40:14.471 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:40:14.076+0000 7f26abc50700 -1 osd.4 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:14.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:14.392+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:14.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:14 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:14.445+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:14.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:14 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:14.524+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:15.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:14.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:15.364 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:15 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:15.069+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:15.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:15 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:15.388+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:15.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:15 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:15.441+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:15.970 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:15 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:15.503+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:16.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:15 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:15.930+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:16.364 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:16.064+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:16.663 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:16.342+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:16.663 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:16.342+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:16.720 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:40:16.392+0000 7f26a8244700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:50.223573+0000 front 2024-04-25T01:39:50.223605+0000 (oldest deadline 2024-04-25T01:40:15.523765+0000) 2024-04-25T01:40:16.721 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:16 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:16.502+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:16.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:16 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:16.399+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:17.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:16.980+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:17.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:16 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:16.980+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:17.363 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:17 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:17.057+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:17.536 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:17 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:17.388+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:17.536 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:17 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:17.388+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:17.614 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:17 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:17.380+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:17.622 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:40:17.345+0000 7f26a8244700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:50.223573+0000 front 2024-04-25T01:39:50.223605+0000 (oldest deadline 2024-04-25T01:40:15.523765+0000) 2024-04-25T01:40:17.622 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:17 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:17.551+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:18.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:18.016+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:18.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:18.016+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:18.364 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:18.015+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:18.586 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4[44086]: 2024-04-25T01:40:18.335+0000 7f26a8244700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:50.223573+0000 front 2024-04-25T01:39:50.223605+0000 (oldest deadline 2024-04-25T01:40:15.523765+0000) 2024-04-25T01:40:18.587 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:18.540+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:18.587 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:18.540+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:18.741 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:18.418+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:18.741 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:18.418+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:18.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:18.408+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:18.864 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:18.408+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:18.882 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:18 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:18.601+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:19.005 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:18.757+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:19.005 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:18.967+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:19.006 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:18 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:18.967+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:19.357 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:19 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:19.009+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:19.358 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:19 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:19.009+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:19.614 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:19 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:19.374+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:19.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:19 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:19.374+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:19.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:19 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:19.390+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:19.756 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:19 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:19.390+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:19.928 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:19.580+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:19.929 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:19.516+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:19.929 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:19 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:19.516+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:19.929 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:19 smithi038 podman[67810]: 2024-04-25 01:40:19.639399559 +0000 UTC m=+5.687440388 container died 4a84b8775b71d29d83741114094cbe9b25b83e51d9d645959a2158cd0248f506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True) 2024-04-25T01:40:20.221 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:20 smithi038 podman[67810]: 2024-04-25 01:40:20.043891436 +0000 UTC m=+6.091932265 container cleanup 4a84b8775b71d29d83741114094cbe9b25b83e51d9d645959a2158cd0248f506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T01:40:20.221 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:20 smithi038 bash[67810]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4 2024-04-25T01:40:20.255 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:19 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:19.800+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:20.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:19 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:19.986+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:20.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:19 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:19.986+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:20.324 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:20 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:20.008+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:20.324 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:20 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:20.008+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:20.596 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:20 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:20.542+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:20.596 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:20 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:20.540+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:20.596 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:20 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:20.540+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:20.596 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:20 smithi038 podman[67839]: 2024-04-25 01:40:20.295258857 +0000 UTC m=+0.650739373 container remove 4a84b8775b71d29d83741114094cbe9b25b83e51d9d645959a2158cd0248f506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T01:40:20.614 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:20 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:20.341+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:20.614 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:20 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:20.341+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:20.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:20 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:20.340+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:20.756 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:20 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:20.340+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:20.877 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:20 smithi038 podman[67932]: 2024-04-25 01:40:20.515784082 +0000 UTC m=+0.018514849 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:20.877 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:20 smithi038 podman[67932]: 2024-04-25 01:40:20.639142603 +0000 UTC m=+0.141873341 container create bc4f32569a7ae085b1a77cd39f3554743c4c7a2509772d78d68077ec6346d93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T01:40:21.221 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:21 smithi038 podman[67932]: 2024-04-25 01:40:21.007252851 +0000 UTC m=+0.509983591 container init bc4f32569a7ae085b1a77cd39f3554743c4c7a2509772d78d68077ec6346d93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:40:21.221 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:21 smithi038 podman[67932]: 2024-04-25 01:40:21.010582328 +0000 UTC m=+0.513313079 container start bc4f32569a7ae085b1a77cd39f3554743c4c7a2509772d78d68077ec6346d93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:40:21.221 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:21 smithi038 podman[67932]: 2024-04-25 01:40:21.065921425 +0000 UTC m=+0.568652180 container attach bc4f32569a7ae085b1a77cd39f3554743c4c7a2509772d78d68077ec6346d93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:40:21.255 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:20 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:20.803+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:21.255 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:20 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:20.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:21.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:20 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:20.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:21.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:20 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:20.970+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:21.332 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:20 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:20.970+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:21.574 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 25 01:40:21 smithi038 podman[67932]: 2024-04-25 01:40:21.323127117 +0000 UTC m=+0.825857860 container died bc4f32569a7ae085b1a77cd39f3554743c4c7a2509772d78d68077ec6346d93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T01:40:21.574 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:21 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:21.565+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:21.574 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:21 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:21.565+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:21.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:21.348+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:21.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:21.348+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:21.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:21 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:21.330+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:21.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:21 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:21.330+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:21.858 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:21 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:21.588+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:22.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T01:40:22.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:22.184 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T01:40:22.184 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T01:40:22.184 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.5 2024-04-25T01:40:22.211 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:21 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:21.827+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:22.211 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:21 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:21.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:22.211 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:21 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:21.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:22.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:21.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:22.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:21.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:22.505 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:22 smithi156 systemd[1]: Stopping Ceph osd.5 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:22.506 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:22.339+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:22.506 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:22.339+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:22.614 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:22.370+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:22.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:22.370+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:22.775 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:22.526+0000 7f9641f2b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:22.775 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:22.526+0000 7f9641f2b700 -1 osd.5 90 *** Got signal Terminated *** 2024-04-25T01:40:22.775 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:22.526+0000 7f9641f2b700 -1 osd.5 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:22.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:22 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:22.631+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:22.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:22 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:22.592+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:22.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:22 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:22.592+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:23.255 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:22.790+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:23.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:22.959+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:23.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:22 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:22.959+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:23.313 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:22.921+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:23.314 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:22 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:22.921+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:23.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:23 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:23.329+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:23.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:23 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:23.329+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:23.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:23 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:23.329+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:23.755 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:23.344+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:23.756 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:23.344+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:23.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:23 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:23.597+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:23.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:23 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:23.580+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:23.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:23 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:23.580+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:24.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:23 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:23.963+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:24.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:23 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:23.963+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:24.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:23 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:23.963+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:24.255 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:23.774+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:24.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:23.991+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:24.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:23.991+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:24.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:23 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:23.991+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:24.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:24.345+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:24.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:24.345+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:24.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:24.345+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:24.710 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:24 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:24.344+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:24.710 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:24 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:24.344+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:24.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:24 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:24.618+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:24.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:24 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:24.618+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:24.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:24 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:24.592+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:24.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:24 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:24.592+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:25.005 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:24 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:24.726+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:25.330 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:24.985+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:25.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:24.985+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:25.331 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:24.985+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:25.339 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:25.028+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:25.339 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:25.028+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:25.339 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:25.028+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:25.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:25 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:25.346+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:25.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:25 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:25.346+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:25.615 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:25 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:25.346+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:25.663 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:25.355+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:25.663 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:25.355+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:25.663 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:25.355+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.268174+0000 front 2024-04-25T01:40:01.268222+0000 (oldest deadline 2024-04-25T01:40:24.768413+0000) 2024-04-25T01:40:25.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:25 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:25.635+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:25.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:25 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:25.635+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:25.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:25 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:25.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:25.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:25 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:25.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:26.005 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:25.678+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:26.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:25 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:25.678+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:26.333 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:26.037+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:26.333 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:26.037+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:26.333 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:26.037+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:26.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:26.018+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:26.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:26.018+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:26.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:26.018+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:26.668 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:26.348+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:26.668 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:26.348+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:26.668 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:26.348+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.268174+0000 front 2024-04-25T01:40:01.268222+0000 (oldest deadline 2024-04-25T01:40:24.768413+0000) 2024-04-25T01:40:26.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:26.386+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:26.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:26.386+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:26.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:26.386+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:26.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:26 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:26.620+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:26.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:26 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:26.620+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:26.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:26 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:26.586+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:26.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:26 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:26.586+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:26.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:26 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:26.586+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:27.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:26.683+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:27.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:26 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:26.683+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:27.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:26.994+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:27.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:26.994+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:27.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:26 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:26.994+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:27.505 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:27.303+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.466525+0000 front 2024-04-25T01:39:45.466569+0000 (oldest deadline 2024-04-25T01:40:09.566446+0000) 2024-04-25T01:40:27.506 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:27.303+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.566970+0000 front 2024-04-25T01:39:49.566906+0000 (oldest deadline 2024-04-25T01:40:15.466898+0000) 2024-04-25T01:40:27.506 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5[44315]: 2024-04-25T01:40:27.303+0000 7f963e51f700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.268174+0000 front 2024-04-25T01:40:01.268222+0000 (oldest deadline 2024-04-25T01:40:24.768413+0000) 2024-04-25T01:40:27.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:27.077+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:27.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:27.077+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:27.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:27.077+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:27.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:27.429+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:27.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:27.429+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:27.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:27.429+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:27.965 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:27.696+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:27.965 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:27 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:27.696+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:27.965 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:27 smithi156 podman[68973]: 2024-04-25 01:40:27.928813672 +0000 UTC m=+5.528399765 container died 245c957b9d597d077917c8450c72d4f5239df0068d8b00be2a21e8fef5de3a64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:40:27.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:27 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:27.617+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:27.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:27 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:27.617+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:27.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:27 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:27.539+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:27.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:27 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:27.539+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:27.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:27 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:27.539+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:28.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:28 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:28.068+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:28.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:28 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:28.068+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:28.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:28 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:28.068+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:28.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:27.954+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:28.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:27.954+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:28.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:27 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:27.954+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:28.595 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:28 smithi156 podman[68973]: 2024-04-25 01:40:28.309458371 +0000 UTC m=+5.909044471 container cleanup 245c957b9d597d077917c8450c72d4f5239df0068d8b00be2a21e8fef5de3a64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:40:28.595 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:28 smithi156 bash[68973]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5 2024-04-25T01:40:28.721 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:28 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:28.527+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:28.721 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:28 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:28.527+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:28.721 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:28 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:28.527+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:28.722 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:28 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:28.623+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:28.722 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:28 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:28.623+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:28.867 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:28.460+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:29.507 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:28.460+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:29.507 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:28.460+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:29.508 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:28 smithi156 podman[68992]: 2024-04-25 01:40:28.611334899 +0000 UTC m=+0.683352398 container remove 245c957b9d597d077917c8450c72d4f5239df0068d8b00be2a21e8fef5de3a64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T01:40:29.509 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:28.973+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:29.509 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:28.973+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:29.509 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:28 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:28.973+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:29.509 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:28 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:28.732+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:29.509 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:28 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:28.732+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:29.510 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:29.078+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:29.510 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:29.078+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:29.510 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:29.078+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:29.510 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:28 smithi156 podman[69086]: 2024-04-25 01:40:28.873705061 +0000 UTC m=+0.018080837 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:29.510 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:28 smithi156 podman[69086]: 2024-04-25 01:40:28.987067527 +0000 UTC m=+0.131443302 container create 5027b5a0e60c99e1e5bbad4d3f4c9b36c804f32cd281c42d71919a6c9c7f7cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T01:40:29.511 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:29 smithi156 podman[69086]: 2024-04-25 01:40:29.231213879 +0000 UTC m=+0.375589655 container init 5027b5a0e60c99e1e5bbad4d3f4c9b36c804f32cd281c42d71919a6c9c7f7cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T01:40:29.511 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:29 smithi156 podman[69086]: 2024-04-25 01:40:29.234511677 +0000 UTC m=+0.378887453 container start 5027b5a0e60c99e1e5bbad4d3f4c9b36c804f32cd281c42d71919a6c9c7f7cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:40:29.511 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:29 smithi156 podman[69086]: 2024-04-25 01:40:29.331268586 +0000 UTC m=+0.475644378 container attach 5027b5a0e60c99e1e5bbad4d3f4c9b36c804f32cd281c42d71919a6c9c7f7cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:40:29.756 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:29 smithi156 podman[69086]: 2024-04-25 01:40:29.520769267 +0000 UTC m=+0.665145043 container died 5027b5a0e60c99e1e5bbad4d3f4c9b36c804f32cd281c42d71919a6c9c7f7cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T01:40:29.756 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:29.751+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:29.756 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:29 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:29.751+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:29.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:29.507+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:29.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:29.507+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:29.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:29.507+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:29.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:29.593+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:29.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:29.593+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:29.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:29.520+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:29.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:29.520+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:29.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:29 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:29.520+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:30.067 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T01:40:30.131 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:30 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:30.067+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:30.131 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:30 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:30.067+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:30.131 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:30 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:30.067+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:30.131 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:30 smithi156 podman[69112]: 2024-04-25 01:40:30.068078385 +0000 UTC m=+0.536114053 container remove 5027b5a0e60c99e1e5bbad4d3f4c9b36c804f32cd281c42d71919a6c9c7f7cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:40:30.132 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:30 smithi156 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-25T01:40:30.132 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:30 smithi156 systemd[1]: Stopped Ceph osd.5 for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:40:30.132 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 01:40:30 smithi156 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.5.service: Consumed 8min 5.256s CPU time. 2024-04-25T01:40:30.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:30.148 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T01:40:30.148 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T01:40:30.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.6 2024-04-25T01:40:30.365 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 systemd[1]: Stopping Ceph osd.6 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:30.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:29.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:30.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:29.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:30.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:29 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:29.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:30.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:30.475+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:30.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:30.475+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:30.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:30.475+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:30.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:30.479+0000 7fdd81dfd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:30.866 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:30.479+0000 7fdd81dfd700 -1 osd.6 90 *** Got signal Terminated *** 2024-04-25T01:40:30.866 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:30.479+0000 7fdd81dfd700 -1 osd.6 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:30.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:30 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:30.575+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:30.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:30 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:30.575+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:30.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:30 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:30.567+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:30.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:30 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:30.567+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:30.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:30 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:30.567+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:31.005 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:30 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:30.733+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:31.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:30 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:30.733+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:31.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:30.995+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:31.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:30.995+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:31.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:30.995+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:31.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:31 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:31.052+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:31.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:31 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:31.052+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:31.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:31 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:31.052+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:31.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:31.480+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:31.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:31.480+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:31.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:31.480+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:31.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:31 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:31.550+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:31.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:31 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:31.550+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:31.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:31 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:31.565+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:31.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:31 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:31.565+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:31.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:31 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:31.565+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:32.005 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:31 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:31.742+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:32.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:31 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:31.742+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:32.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:31.995+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:32.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:31.995+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:32.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:31 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:31.995+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:32.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:32 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:32.078+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:32.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:32 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:32.078+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:32.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:32 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:32.078+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:32.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:32 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:32.449+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:32.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:32 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:32.449+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:32.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:32 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:32.449+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:32.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:32 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:32.512+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:32.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:32 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:32.512+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:32.971 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:32 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:32.536+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:32.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:32 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:32.536+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:32.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:32 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:32.536+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:33.099 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:32 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:32.785+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:33.099 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:32 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:32.785+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:33.100 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:32 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:32.785+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:33.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:32 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:32.964+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:33.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:32 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:32.964+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:33.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:32 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:32.964+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:33.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:33.114+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:33.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:33.114+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:33.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:33.114+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:33.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:33.114+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:33.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:33.441+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:33.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:33.441+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:33.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:33.441+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:33.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:33.441+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.092244+0000 front 2024-04-25T01:40:09.092274+0000 (oldest deadline 2024-04-25T01:40:32.592374+0000) 2024-04-25T01:40:33.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:33.529+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:33.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:33.529+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:33.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:33.562+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:33.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:33.562+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:33.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:33.562+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:33.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:33.562+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:34.101 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:33.811+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:34.101 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:33.811+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:34.101 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:33 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:33.811+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:34.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:33.963+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:34.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:33.963+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:34.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:33 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:33.963+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:34.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:34.116+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:34.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:34.116+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:34.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:34.116+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:34.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:34.116+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:34.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:34.461+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.087943+0000 front 2024-04-25T01:39:44.087798+0000 (oldest deadline 2024-04-25T01:40:07.588029+0000) 2024-04-25T01:40:34.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:34.461+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:49.289160+0000 front 2024-04-25T01:39:49.288986+0000 (oldest deadline 2024-04-25T01:40:17.489811+0000) 2024-04-25T01:40:34.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:34.461+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.591063+0000 front 2024-04-25T01:39:58.591014+0000 (oldest deadline 2024-04-25T01:40:23.291022+0000) 2024-04-25T01:40:34.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6[56893]: 2024-04-25T01:40:34.461+0000 7fdd7e3f1700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.092244+0000 front 2024-04-25T01:40:09.092274+0000 (oldest deadline 2024-04-25T01:40:32.592374+0000) 2024-04-25T01:40:34.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:34.559+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:34.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:34.559+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:34.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:34.559+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:34.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:34.574+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:34.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:34.574+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:34.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:34.574+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:34.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:34 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:34.574+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:35.129 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:34.814+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:35.129 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:34.814+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:35.129 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:34 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:34.814+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:35.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:34.977+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:35.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:34.977+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:35.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:34 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:34.977+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:35.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:35.143+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:35.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:35.143+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:35.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:35.143+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:35.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:35.143+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:35.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:35.546+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:35.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:35.546+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:35.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:35.546+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:35.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:35.610+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:35.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:35.610+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:35.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:35.610+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:35.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:35 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:35.610+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:36.123 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:35.788+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:36.123 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:35.788+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:36.123 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:35 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:35.788+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:36.223 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:35 smithi016 podman[94822]: 2024-04-25 01:40:35.913316615 +0000 UTC m=+5.549771335 container stop 744b86c5ff2e60ded4be0b0a9037aef4991b710c5b8441ffe445d1259e1bcdf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T01:40:36.224 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:35 smithi016 podman[94822]: 2024-04-25 01:40:35.941037413 +0000 UTC m=+5.577492134 container died 744b86c5ff2e60ded4be0b0a9037aef4991b710c5b8441ffe445d1259e1bcdf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T01:40:36.224 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:35 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:35.942+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:36.224 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:35 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:35.942+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:36.224 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:35 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:35.942+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:36.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:36.137+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:36.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:36.137+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:36.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:36.137+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:36.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:36.137+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:36.557 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:36 smithi016 podman[94822]: 2024-04-25 01:40:36.363355236 +0000 UTC m=+5.999809956 container cleanup 744b86c5ff2e60ded4be0b0a9037aef4991b710c5b8441ffe445d1259e1bcdf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T01:40:36.557 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:36 smithi016 bash[94822]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6 2024-04-25T01:40:36.865 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:36 smithi016 podman[94840]: 2024-04-25 01:40:36.573186497 +0000 UTC m=+0.649886548 container remove 744b86c5ff2e60ded4be0b0a9037aef4991b710c5b8441ffe445d1259e1bcdf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:40:36.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:36.517+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:36.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:36.517+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:36.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:36.517+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:36.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:36.616+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:36.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:36.616+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:36.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:36.616+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:36.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:36 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:36.616+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:37.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:36.758+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:37.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:36.758+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:37.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:36 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:36.758+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:37.154 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:36 smithi016 podman[94927]: 2024-04-25 01:40:36.893374936 +0000 UTC m=+0.018693192 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:37.155 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:37 smithi016 podman[94927]: 2024-04-25 01:40:37.015790893 +0000 UTC m=+0.141109168 container create 7b6ac4f355de5f80a450b199a4bdf22f2f1ad49f2c3ced0aaffce06214176b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:40:37.155 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:36.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:37.155 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:36.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:37.155 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:36.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:37.155 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:36 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:36.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:37.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:37.153+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:37.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:37.153+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:37.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:37.153+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:37.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:37.153+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:37.598 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:37 smithi016 podman[94927]: 2024-04-25 01:40:37.302057084 +0000 UTC m=+0.427375323 container init 7b6ac4f355de5f80a450b199a4bdf22f2f1ad49f2c3ced0aaffce06214176b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:40:37.598 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:37 smithi016 podman[94927]: 2024-04-25 01:40:37.305287504 +0000 UTC m=+0.430605740 container start 7b6ac4f355de5f80a450b199a4bdf22f2f1ad49f2c3ced0aaffce06214176b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:40:37.598 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:37 smithi016 podman[94927]: 2024-04-25 01:40:37.401898976 +0000 UTC m=+0.527217221 container attach 7b6ac4f355de5f80a450b199a4bdf22f2f1ad49f2c3ced0aaffce06214176b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:37.599 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 25 01:40:37 smithi016 podman[94927]: 2024-04-25 01:40:37.614043398 +0000 UTC m=+0.739361637 container died 7b6ac4f355de5f80a450b199a4bdf22f2f1ad49f2c3ced0aaffce06214176b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:40:37.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:37.563+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:37.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:37.563+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:37.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:37.563+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:37.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:37.614+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:37.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:37.614+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:37.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:37.614+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:37.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:37 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:37.614+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:38.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:37.743+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:38.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:37.743+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:38.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:37 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:37.743+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:38.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:37.891+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:38.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:37.891+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:38.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:37.891+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:38.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:37 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:37.891+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:38.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T01:40:38.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:38.484 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T01:40:38.484 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T01:40:38.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.7 2024-04-25T01:40:38.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:38.171+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:38.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:38.171+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:38.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:38.171+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:38.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:38.171+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:38.521+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:38.521+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:38.521+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 systemd[1]: Stopping Ceph osd.7 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:38.596+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:38.596+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:38.596+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:38.760 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:38.596+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:39.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:38.716+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:39.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:38.716+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:39.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:38 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:38.716+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:39.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:38 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:38.893+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:39.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:38 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:38.893+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:39.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:38 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:38.893+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:39.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:38 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:38.893+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:39.221 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:38.773+0000 7f428c825700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:39.221 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:38.773+0000 7f428c825700 -1 osd.7 90 *** Got signal Terminated *** 2024-04-25T01:40:39.221 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:38.773+0000 7f428c825700 -1 osd.7 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:39.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:39.140+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:39.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:39.140+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:39.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:39.140+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:39.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:39.140+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:39.568+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:39.568+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:39.568+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:39.601+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:39.601+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:39.601+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:39.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:39 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:39.601+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:40.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:39.703+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:40.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:39.703+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:40.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:39 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:39.703+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:40.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:39 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:39.923+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:40.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:39 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:39.923+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:40.115 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:39 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:39.923+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:40.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:39 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:39.923+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:40.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:40.121+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:40.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:40.121+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:40.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:40.121+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:40.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:40.121+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:40.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:40.121+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:40.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:40.589+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:40.589+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:40.589+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:40.568+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:40.568+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:40.568+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:40.568+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:40.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:40 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:40.568+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:41.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:40.697+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:41.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:40.697+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:41.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:40.697+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:41.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:40 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:40.697+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:41.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:40 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:40.956+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:41.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:40 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:40.956+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:41.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:40 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:40.956+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:41.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:40 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:40.956+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:41.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:41.102+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:41.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:41.102+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:41.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:41.102+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:41.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:41.102+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:41.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:41.102+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:41.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:41.601+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:41.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:41.601+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:41.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:41.601+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:41.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:41.601+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.060520+0000 front 2024-04-25T01:40:16.060546+0000 (oldest deadline 2024-04-25T01:40:40.760723+0000) 2024-04-25T01:40:41.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:41.525+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:41.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:41.525+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:41.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:41.525+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:41.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:41.525+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:41.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:41 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:41.525+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:42.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:41.667+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:42.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:41.667+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:42.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:41.667+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:42.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:41 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:41.667+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:42.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:41.943+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:42.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:41.943+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:42.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:41.943+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:42.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:41.943+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:42.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:41 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:41.943+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:42.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:42.106+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:42.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:42.106+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:42.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:42.106+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:42.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:42.106+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:42.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:42.106+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:42.553+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:42.553+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:42.553+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:42.553+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.060520+0000 front 2024-04-25T01:40:16.060546+0000 (oldest deadline 2024-04-25T01:40:40.760723+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:42.575+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:42.575+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:42.575+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:42.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:42.575+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:42.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:42.575+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:43.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:42.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:43.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:42.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:43.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:42.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:43.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:42 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:42.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:43.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:42 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:42.901+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:43.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:42 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:42.901+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:43.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:42 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:42.901+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:43.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:42 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:42.901+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:43.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:42 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:42.901+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:43.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:43.099+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:43.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:43.099+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:43.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:43.099+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:43.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:43.099+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:43.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:43.099+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:43.506+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.855198+0000 front 2024-04-25T01:39:52.855228+0000 (oldest deadline 2024-04-25T01:40:18.155407+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:43.506+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:02.157283+0000 front 2024-04-25T01:40:02.157249+0000 (oldest deadline 2024-04-25T01:40:23.857341+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:43.506+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.659436+0000 front 2024-04-25T01:40:09.659462+0000 (oldest deadline 2024-04-25T01:40:33.759463+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7[50649]: 2024-04-25T01:40:43.506+0000 7f4288e19700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.060520+0000 front 2024-04-25T01:40:16.060546+0000 (oldest deadline 2024-04-25T01:40:40.760723+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:43.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:43.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:43.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:43.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:43.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:43 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:43.598+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:44.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:43.678+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:44.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:43.678+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:44.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:43.678+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:44.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:43 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:43.678+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:44.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:43 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:43.882+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:44.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:43 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:43.882+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:44.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:43 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:43.882+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:44.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:43 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:43.882+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:44.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:43 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:43.882+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:44.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:44.061+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:44.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:44.061+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:44.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:44.061+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:44.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:44.062+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:44.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:44.062+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:44.644 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:44 smithi038 podman[68039]: 2024-04-25 01:40:44.382583829 +0000 UTC m=+5.689118419 container died da18fb9c9517fcf87aee88ccc7b7226fe5ec0b8e327e4bf3bf4d1cfb85b161a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T01:40:44.644 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:44 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:44.599+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:44.644 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:44 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:44.599+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:44.644 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:44 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:44.599+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:44.644 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:44 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:44.599+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:44.644 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:44 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:44.599+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:44.971 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:44 smithi038 podman[68039]: 2024-04-25 01:40:44.766690798 +0000 UTC m=+6.073225379 container cleanup da18fb9c9517fcf87aee88ccc7b7226fe5ec0b8e327e4bf3bf4d1cfb85b161a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:44.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:44 smithi038 bash[68039]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7 2024-04-25T01:40:45.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:44.634+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:45.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:44.634+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:45.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:44.634+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:45.006 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:44 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:44.634+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:45.312 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68056]: 2024-04-25 01:40:45.018238835 +0000 UTC m=+0.630474356 container remove da18fb9c9517fcf87aee88ccc7b7226fe5ec0b8e327e4bf3bf4d1cfb85b161a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T01:40:45.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:44 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:44.926+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:45.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:44 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:44.926+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:45.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:44 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:44.926+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:45.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:44 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:44.926+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:45.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:44 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:44.926+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:45.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:45.049+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:45.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:45.049+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:45.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:45.049+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:45.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:45.049+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:45.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:45.049+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:45.617 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68145]: 2024-04-25 01:40:45.32748281 +0000 UTC m=+0.113828304 container create 72f5b2a4b2fbbd05b9a2447b7eb9ddc13ad7b8cba62cb2eab000a8d3c46164cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:45.617 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68145]: 2024-04-25 01:40:45.232781662 +0000 UTC m=+0.019127171 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:45.617 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68145]: 2024-04-25 01:40:45.551353019 +0000 UTC m=+0.337698507 container init 72f5b2a4b2fbbd05b9a2447b7eb9ddc13ad7b8cba62cb2eab000a8d3c46164cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T01:40:45.618 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68145]: 2024-04-25 01:40:45.554704969 +0000 UTC m=+0.341050459 container start 72f5b2a4b2fbbd05b9a2447b7eb9ddc13ad7b8cba62cb2eab000a8d3c46164cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68145]: 2024-04-25 01:40:45.664006424 +0000 UTC m=+0.450351920 container attach 72f5b2a4b2fbbd05b9a2447b7eb9ddc13ad7b8cba62cb2eab000a8d3c46164cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 25 01:40:45 smithi038 podman[68145]: 2024-04-25 01:40:45.868090815 +0000 UTC m=+0.654436313 container died 72f5b2a4b2fbbd05b9a2447b7eb9ddc13ad7b8cba62cb2eab000a8d3c46164cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:45 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:45.632+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:45 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:45.632+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:45 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:45.632+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:45 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:45.632+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:45.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:45 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:45.632+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:46.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:45.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:46.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:45.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:46.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:45.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:46.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:45 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:45.642+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:46.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:45 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:45.931+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:46.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:45 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:45.931+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:46.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:45 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:45.931+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:46.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:45 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:45.931+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:46.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:45 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:45.931+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:46.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T01:40:46.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:46.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:46.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:46.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:46.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:46.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:46.506 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:46.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:46.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:46.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:46.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:46.531 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T01:40:46.532 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T01:40:46.532 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.8 2024-04-25T01:40:46.722 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:46.618+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:46.722 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:46.618+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:46.722 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:46.618+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:46.722 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:46.618+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:46.722 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:46.618+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:46.841 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 systemd[1]: Stopping Ceph osd.8 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:46.841 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.623+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:46.841 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.623+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:46.841 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.623+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:46.841 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.623+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:47.256 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.854+0000 7fbe30c46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:47.256 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.854+0000 7fbe30c46700 -1 osd.8 90 *** Got signal Terminated *** 2024-04-25T01:40:47.256 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:46 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:46.854+0000 7fbe30c46700 -1 osd.8 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:47.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:47.026+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:47.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:47.026+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:47.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:47.026+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:47.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:47.026+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:47.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:47.026+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:47.365 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:46 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:46.903+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:47.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:46 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:46.903+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:47.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:46 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:46.903+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:47.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:46 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:46.903+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:47.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:46 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:46.903+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:47.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:47 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:47.608+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:47.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:47 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:47.608+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:47.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:47 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:47.608+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:47.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:47 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:47.608+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:47.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:47 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:47.608+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:48.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:47.636+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:48.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:47.636+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:48.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:47.636+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:48.007 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:47 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:47.636+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:48.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:47 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:47.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:48.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:47 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:47.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:48.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:47 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:47.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:48.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:47 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:47.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:48.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:47 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:47.918+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:48.446 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:48.446 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:48.446 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:48.446 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:48.447 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.047+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:48.756 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:48.605+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:48.757 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:48.605+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:48.757 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:48.605+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:48.757 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:48.605+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:48.757 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:48 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:48.605+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:25.653478+0000 front 2024-04-25T01:40:25.653515+0000 (oldest deadline 2024-04-25T01:40:48.553595+0000) 2024-04-25T01:40:48.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:48 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:48.634+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:48.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:48 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:48.634+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:48.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:48 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:48.634+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:48.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:48 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:48.634+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:48.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:48 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:48.634+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:49.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.998+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:49.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.998+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:49.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.998+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:49.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.998+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:49.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:48.998+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:49.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:48 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:48.961+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:49.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:48 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:48.961+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:49.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:48 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:48.961+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:49.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:48 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:48.961+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:49.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:48 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:48.961+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:49.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:49.643+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:49.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:49.643+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:49.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:49.643+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:49.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:49.643+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:49.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:49 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:49.643+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:50.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:49.595+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:50.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:49.595+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:50.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:49.595+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:50.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:49.595+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:50.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:49 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:49.595+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:25.653478+0000 front 2024-04-25T01:40:25.653515+0000 (oldest deadline 2024-04-25T01:40:48.553595+0000) 2024-04-25T01:40:50.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:49 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:49.940+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:50.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:49 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:49.940+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:50.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:49 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:49.940+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:50.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:49 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:49.940+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:50.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:49 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:49.940+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:50.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.013+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:50.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.013+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:50.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.013+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:50.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.013+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:50.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.013+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:50.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.013+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:50.959 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:50.590+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:50.959 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:50.590+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:50.959 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:50.590+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:50.960 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:50.590+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:50.960 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:50.590+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:25.653478+0000 front 2024-04-25T01:40:25.653515+0000 (oldest deadline 2024-04-25T01:40:48.553595+0000) 2024-04-25T01:40:50.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:50 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:50.607+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:50.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:50 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:50.607+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:50.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:50 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:50.607+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:50.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:50 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:50.607+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:50.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:50 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:50.607+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:51.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.973+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:51.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.973+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:51.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.973+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:51.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.973+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:51.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.973+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:51.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:50.973+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:51.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:50 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:50.967+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:51.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:50 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:50.967+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:51.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:50 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:50.967+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:51.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:50 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:50.967+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:51.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:50 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:50.967+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:51.913 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:51.625+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:53.247555+0000 front 2024-04-25T01:39:53.247692+0000 (oldest deadline 2024-04-25T01:40:18.547470+0000) 2024-04-25T01:40:51.913 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:51.625+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:00.149167+0000 front 2024-04-25T01:40:00.149294+0000 (oldest deadline 2024-04-25T01:40:25.449287+0000) 2024-04-25T01:40:51.913 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:51.625+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.550322+0000 front 2024-04-25T01:40:09.550488+0000 (oldest deadline 2024-04-25T01:40:32.350908+0000) 2024-04-25T01:40:51.913 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:51.625+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:15.251850+0000 front 2024-04-25T01:40:15.251950+0000 (oldest deadline 2024-04-25T01:40:39.951770+0000) 2024-04-25T01:40:51.914 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8[51293]: 2024-04-25T01:40:51.625+0000 7fbe2d23a700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:25.653478+0000 front 2024-04-25T01:40:25.653515+0000 (oldest deadline 2024-04-25T01:40:48.553595+0000) 2024-04-25T01:40:51.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:51 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:51.612+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:51.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:51 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:51.612+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:51.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:51 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:51.612+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:51.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:51 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:51.612+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:51.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:51 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:51.612+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:51.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:51 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:51.612+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:52.242 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:51.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:52.242 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:51.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:52.242 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:51.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:52.242 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:51.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:52.242 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:51.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:52.242 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:51 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:51.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:52.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:51.935+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:52.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:51.935+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:52.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:51.935+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:52.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:51.935+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:52.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:51 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:51.935+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:52.506 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:52 smithi156 podman[69192]: 2024-04-25 01:40:52.25706718 +0000 UTC m=+5.521025967 container died 961a9a329c8a12e1ff77a63deff9d3877a921959962058abc478885c6d5b0f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T01:40:52.775 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:52 smithi156 podman[69192]: 2024-04-25 01:40:52.605602665 +0000 UTC m=+5.869561458 container cleanup 961a9a329c8a12e1ff77a63deff9d3877a921959962058abc478885c6d5b0f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T01:40:52.775 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:52 smithi156 bash[69192]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8 2024-04-25T01:40:52.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:52 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:52.654+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:52.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:52 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:52.654+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:52.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:52 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:52.654+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:52.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:52 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:52.654+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:52.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:52 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:52.654+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:52.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:52 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:52.654+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:53.206 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:52 smithi156 podman[69221]: 2024-04-25 01:40:52.790236206 +0000 UTC m=+0.526979917 container remove 961a9a329c8a12e1ff77a63deff9d3877a921959962058abc478885c6d5b0f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:40:53.207 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:52 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:52.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:53.207 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:52 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:52.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:53.207 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:52 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:52.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:53.207 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:52 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:52.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:53.207 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:52 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:52.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:53.207 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:52 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:52.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:53.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:52.895+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:53.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:52.895+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:53.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:52.895+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:53.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:52.895+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:53.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:52 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:52.895+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:53.506 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:53 smithi156 podman[69314]: 2024-04-25 01:40:53.125202545 +0000 UTC m=+0.018404966 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:40:53.507 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:53 smithi156 podman[69314]: 2024-04-25 01:40:53.232660687 +0000 UTC m=+0.125863090 container create 224a9a6a69cffe9e3cd28db91b3224326521262447590e57ec533280e16cc8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T01:40:53.507 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:53 smithi156 podman[69314]: 2024-04-25 01:40:53.44370261 +0000 UTC m=+0.336905014 container init 224a9a6a69cffe9e3cd28db91b3224326521262447590e57ec533280e16cc8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T01:40:53.507 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:53 smithi156 podman[69314]: 2024-04-25 01:40:53.447029989 +0000 UTC m=+0.340232405 container start 224a9a6a69cffe9e3cd28db91b3224326521262447590e57ec533280e16cc8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:40:53.970 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:53 smithi156 podman[69314]: 2024-04-25 01:40:53.522905324 +0000 UTC m=+0.416107747 container attach 224a9a6a69cffe9e3cd28db91b3224326521262447590e57ec533280e16cc8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:40:53.971 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 01:40:53 smithi156 podman[69314]: 2024-04-25 01:40:53.734076934 +0000 UTC m=+0.627279337 container died 224a9a6a69cffe9e3cd28db91b3224326521262447590e57ec533280e16cc8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:40:53.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:53 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:53.644+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:53.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:53 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:53.644+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:53.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:53 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:53.644+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:53.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:53 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:53.644+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:53.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:53 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:53.644+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:53.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:53 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:53.644+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:54.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:53.984+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:54.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:53.984+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:54.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:53.984+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:54.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:53.984+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:54.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:53.984+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:54.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:53 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:53.984+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:54.328 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T01:40:54.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:40:54.365 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T01:40:54.366 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T01:40:54.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.9 2024-04-25T01:40:54.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:53.932+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:54.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:53.932+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:54.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:53.932+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:54.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:53.932+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:54.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:53.932+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:54.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:53 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:53.932+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:27.148368+0000 front 2024-04-25T01:40:27.148246+0000 (oldest deadline 2024-04-25T01:40:52.948734+0000) 2024-04-25T01:40:54.703 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 systemd[1]: Stopping Ceph osd.9 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:40:54.703 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.716+0000 7f5007ef1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:40:54.703 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.716+0000 7f5007ef1700 -1 osd.9 90 *** Got signal Terminated *** 2024-04-25T01:40:54.703 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.716+0000 7f5007ef1700 -1 osd.9 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:40:54.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:54.658+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:54.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:54.658+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:54.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:54.658+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:54.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:54.658+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:54.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:54.658+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:54.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:54 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:54.658+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:55.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:55.116 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:55.117 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:55.117 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:55.117 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:55.117 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:54 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:54.960+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:27.148368+0000 front 2024-04-25T01:40:27.148246+0000 (oldest deadline 2024-04-25T01:40:52.948734+0000) 2024-04-25T01:40:55.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:54.992+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:55.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:54.992+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:55.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:54.992+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:55.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:54.992+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:55.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:54.992+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:55.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:54.992+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:55.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:55 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:55.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:55.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:55 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:55.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:55.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:55 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:55.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:55.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:55 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:55.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:55.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:55 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:55.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:55.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:55 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:55.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:56.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.012+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:56.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.012+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:56.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.012+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:56.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.012+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:56.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.012+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:56.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.012+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:56.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:55.981+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:56.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:55.981+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:56.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:55.981+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:56.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:55.981+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:56.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:55.981+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:56.367 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:55 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:55.981+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:27.148368+0000 front 2024-04-25T01:40:27.148246+0000 (oldest deadline 2024-04-25T01:40:52.948734+0000) 2024-04-25T01:40:56.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:56.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:56.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:56.974 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:56.974 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:56.975 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:56.975 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:56 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:56.689+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:40:57.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.963+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:57.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.963+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:57.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.963+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:57.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.963+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:57.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.963+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:57.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:56 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:56.963+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:57.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:56.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:57.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:56.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:57.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:56.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:57.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:56.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:57.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:56.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:57.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:56 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:56.992+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:27.148368+0000 front 2024-04-25T01:40:27.148246+0000 (oldest deadline 2024-04-25T01:40:52.948734+0000) 2024-04-25T01:40:57.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:57.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:57.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:57.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:57.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:57.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:57.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:57 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:57.690+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:40:58.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:57.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:58.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:57.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:58.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:57.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:58.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:57.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:58.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:57.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:58.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:57 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:57.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:58.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:58.022+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:58.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:58.022+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:58.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:58.022+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:58.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:58.022+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:58.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:58.022+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:58.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:58 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:58.022+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:27.148368+0000 front 2024-04-25T01:40:27.148246+0000 (oldest deadline 2024-04-25T01:40:52.948734+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:40:58.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:58 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:58.701+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:40:59.256 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:58.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:40:59.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:58.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:40:59.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:58.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:40:59.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:58.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:40:59.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:58.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:40:59.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:40:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:40:58.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:43.940922+0000 front 2024-04-25T01:39:43.940964+0000 (oldest deadline 2024-04-25T01:40:09.241136+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:52.742257+0000 front 2024-04-25T01:39:52.742283+0000 (oldest deadline 2024-04-25T01:40:18.642343+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:58.642972+0000 front 2024-04-25T01:39:58.642996+0000 (oldest deadline 2024-04-25T01:40:23.342991+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:10.646475+0000 front 2024-04-25T01:40:10.646564+0000 (oldest deadline 2024-04-25T01:40:36.546532+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.547066+0000 front 2024-04-25T01:40:16.547151+0000 (oldest deadline 2024-04-25T01:40:41.246911+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:27.148368+0000 front 2024-04-25T01:40:27.148246+0000 (oldest deadline 2024-04-25T01:40:52.948734+0000) 2024-04-25T01:40:59.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:40:59 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9[63900]: 2024-04-25T01:40:59.004+0000 7f50004dd700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:33.449809+0000 front 2024-04-25T01:40:33.449838+0000 (oldest deadline 2024-04-25T01:40:58.149878+0000) 2024-04-25T01:40:59.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.656+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:00.222 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:00.223 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:00.223 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:00.223 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:00.223 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:00.223 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:40:59 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:40:59.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:00.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.008+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:00.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.008+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:00.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.008+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:00.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.008+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:00.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.008+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:00.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.008+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:00.328 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:00 smithi016 podman[95044]: 2024-04-25 01:41:00.08544062 +0000 UTC m=+5.497990428 container died ab6658352a4f8ab65c9fe406633b35f26ee72a5ec929104a42f1d5e2f5d6f68b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9) 2024-04-25T01:41:00.616 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:00 smithi016 podman[95044]: 2024-04-25 01:41:00.42524435 +0000 UTC m=+5.837794080 container cleanup ab6658352a4f8ab65c9fe406633b35f26ee72a5ec929104a42f1d5e2f5d6f68b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T01:41:00.616 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:00 smithi016 bash[95044]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9 2024-04-25T01:41:00.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:00.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:00.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:00.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:00.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:00.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:00.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:00 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:00.685+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:01.062 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:00 smithi016 podman[95063]: 2024-04-25 01:41:00.643031263 +0000 UTC m=+0.551037324 container remove ab6658352a4f8ab65c9fe406633b35f26ee72a5ec929104a42f1d5e2f5d6f68b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:01.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.989+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:01.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.989+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:01.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.989+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:01.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.989+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:01.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.989+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:01.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:00 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:00.989+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:01.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:01 smithi016 podman[95152]: 2024-04-25 01:41:00.979581453 +0000 UTC m=+0.018047647 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:41:01.366 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:01 smithi016 podman[95152]: 2024-04-25 01:41:01.077016968 +0000 UTC m=+0.115483144 container create 32310ef0586d5f5a1276c989910cf16f453d66da6cb1301f6f9162b43f24c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T01:41:01.684 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:01 smithi016 podman[95152]: 2024-04-25 01:41:01.387831114 +0000 UTC m=+0.426297291 container init 32310ef0586d5f5a1276c989910cf16f453d66da6cb1301f6f9162b43f24c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T01:41:01.684 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:01 smithi016 podman[95152]: 2024-04-25 01:41:01.391050594 +0000 UTC m=+0.429516769 container start 32310ef0586d5f5a1276c989910cf16f453d66da6cb1301f6f9162b43f24c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:41:01.684 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:01 smithi016 podman[95152]: 2024-04-25 01:41:01.471109681 +0000 UTC m=+0.509575875 container attach 32310ef0586d5f5a1276c989910cf16f453d66da6cb1301f6f9162b43f24c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T01:41:01.959 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 25 01:41:01 smithi016 podman[95152]: 2024-04-25 01:41:01.699216427 +0000 UTC m=+0.737682604 container died 32310ef0586d5f5a1276c989910cf16f453d66da6cb1301f6f9162b43f24c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T01:41:01.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:01.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:01.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:01.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:01.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:01.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:01.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:01 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:01.663+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:02.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T01:41:02.329 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:02.329 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T01:41:02.330 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T01:41:02.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.10 2024-04-25T01:41:02.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:02.022+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:02.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:02.022+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:02.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:02.022+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:02.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:02.022+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:02.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:02.022+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:02.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:02.022+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:02.664 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 systemd[1]: Stopping Ceph osd.10 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:02.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:02.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:02.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:02.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:02.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:02.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:02.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.678+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:02.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.686+0000 7f60c3d03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:02.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.686+0000 7f60c3d03700 -1 osd.10 90 *** Got signal Terminated *** 2024-04-25T01:41:02.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:02 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:02.686+0000 7f60c3d03700 -1 osd.10 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:41:03.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.031+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:03.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.031+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:03.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.031+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:03.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.031+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:03.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.031+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:03.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.031+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:03.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:03.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:03.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:03.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:03.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:03.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:03.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:03 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:03.715+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:04.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:04.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:04.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:04.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:04.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.987+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:04.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:03 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:03.988+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:04.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:04.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:04.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:04.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:04.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:04.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:04.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:04 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:04.671+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:05.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:04 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:04.943+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:05.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:05.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:05.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:05.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:05.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:05.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:05.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:05 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:05.640+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:06.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:05 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:05.908+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:06.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:06.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:06.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:06.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:06.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:06.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:06.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:06 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:06.688+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:07.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:06 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:06.906+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:07.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:44.888317+0000 front 2024-04-25T01:39:44.888354+0000 (oldest deadline 2024-04-25T01:40:09.588499+0000) 2024-04-25T01:41:07.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.289569+0000 front 2024-04-25T01:39:54.289535+0000 (oldest deadline 2024-04-25T01:40:17.789697+0000) 2024-04-25T01:41:07.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:40:01.791685+0000 front 2024-04-25T01:40:01.791856+0000 (oldest deadline 2024-04-25T01:40:26.491835+0000) 2024-04-25T01:41:07.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:08.192956+0000 front 2024-04-25T01:40:08.193201+0000 (oldest deadline 2024-04-25T01:40:32.793491+0000) 2024-04-25T01:41:07.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:18.594724+0000 front 2024-04-25T01:40:18.594758+0000 (oldest deadline 2024-04-25T01:40:40.294871+0000) 2024-04-25T01:41:07.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:26.096237+0000 front 2024-04-25T01:40:26.096315+0000 (oldest deadline 2024-04-25T01:40:50.796367+0000) 2024-04-25T01:41:07.973 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:07 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10[57656]: 2024-04-25T01:41:07.657+0000 7f60c02f7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-25T01:40:30.796869+0000 front 2024-04-25T01:40:30.796855+0000 (oldest deadline 2024-04-25T01:40:56.096736+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:08.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:07 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:07.927+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:08.532 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:08 smithi038 podman[68263]: 2024-04-25 01:41:08.29009768 +0000 UTC m=+5.744684488 container died 4ee1970d7f7aaf68daf56ad01043625c68fec8a8c31abfca3c60f4119632978e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:08.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:08 smithi038 podman[68263]: 2024-04-25 01:41:08.762704835 +0000 UTC m=+6.217291633 container cleanup 4ee1970d7f7aaf68daf56ad01043625c68fec8a8c31abfca3c60f4119632978e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T01:41:08.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:08 smithi038 bash[68263]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10 2024-04-25T01:41:09.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:09.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:09.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:09.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:09.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:09.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:09.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:08 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:08.949+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:09.379 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68280]: 2024-04-25 01:41:09.038838506 +0000 UTC m=+0.747771307 container remove 4ee1970d7f7aaf68daf56ad01043625c68fec8a8c31abfca3c60f4119632978e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T01:41:09.661 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68373]: 2024-04-25 01:41:09.297445451 +0000 UTC m=+0.018765195 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:41:09.661 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68373]: 2024-04-25 01:41:09.406170137 +0000 UTC m=+0.127489868 container create c338af6b8807b9422561615c2580f771ac97f8ab178798c8d061e92663666ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T01:41:09.661 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68373]: 2024-04-25 01:41:09.60050176 +0000 UTC m=+0.321821491 container init c338af6b8807b9422561615c2580f771ac97f8ab178798c8d061e92663666ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:09.661 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68373]: 2024-04-25 01:41:09.603724554 +0000 UTC m=+0.325044287 container start c338af6b8807b9422561615c2580f771ac97f8ab178798c8d061e92663666ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:09.661 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68373]: 2024-04-25 01:41:09.675355863 +0000 UTC m=+0.396675608 container attach c338af6b8807b9422561615c2580f771ac97f8ab178798c8d061e92663666ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-25T01:41:09.972 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 25 01:41:09 smithi038 podman[68373]: 2024-04-25 01:41:09.908817308 +0000 UTC m=+0.630137046 container died c338af6b8807b9422561615c2580f771ac97f8ab178798c8d061e92663666ed3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:10.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:09 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:09.903+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:10.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T01:41:10.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:10.505 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T01:41:10.505 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T01:41:10.505 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.11 2024-04-25T01:41:10.810 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 systemd[1]: Stopping Ceph osd.11 for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:11.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.824+0000 7f3af1e09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.824+0000 7f3af1e09700 -1 osd.11 90 *** Got signal Terminated *** 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.824+0000 7f3af1e09700 -1 osd.11 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:11.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:10 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:10.878+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:12.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:12.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:12.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:12.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:12.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:12.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:12.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:11 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:11.870+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:13.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:13.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:13.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:13.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:13.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:13.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:13.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:12 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:12.919+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:14.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:14.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:14.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:14.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:14.257 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:14.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:14.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:13 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:13.932+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-25T01:39:45.880716+0000 front 2024-04-25T01:39:45.880776+0000 (oldest deadline 2024-04-25T01:40:09.281349+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-25T01:39:54.383295+0000 front 2024-04-25T01:39:54.383144+0000 (oldest deadline 2024-04-25T01:40:16.083162+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T01:39:57.184529+0000 front 2024-04-25T01:39:57.184343+0000 (oldest deadline 2024-04-25T01:40:23.084471+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-25T01:40:09.386627+0000 front 2024-04-25T01:40:09.386647+0000 (oldest deadline 2024-04-25T01:40:32.886533+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-25T01:40:16.288375+0000 front 2024-04-25T01:40:16.288292+0000 (oldest deadline 2024-04-25T01:40:39.788156+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T01:40:23.789656+0000 front 2024-04-25T01:40:23.789622+0000 (oldest deadline 2024-04-25T01:40:49.089596+0000) 2024-04-25T01:41:15.258 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:14 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11[58361]: 2024-04-25T01:41:14.915+0000 7f3aea3f5700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-25T01:40:42.392072+0000 front 2024-04-25T01:40:42.392104+0000 (oldest deadline 2024-04-25T01:41:04.592228+0000) 2024-04-25T01:41:16.507 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:16 smithi156 podman[69419]: 2024-04-25 01:41:16.248136574 +0000 UTC m=+5.540130970 container died 15bf8474b8f93ef22515504ea708fc9e46b91cc3e46a3a97aab5f6038960af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:41:17.007 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:16 smithi156 podman[69419]: 2024-04-25 01:41:16.563475297 +0000 UTC m=+5.855469647 container cleanup 15bf8474b8f93ef22515504ea708fc9e46b91cc3e46a3a97aab5f6038960af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T01:41:17.007 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:16 smithi156 bash[69419]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11 2024-04-25T01:41:17.007 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:16 smithi156 podman[69438]: 2024-04-25 01:41:16.767672426 +0000 UTC m=+0.513667741 container remove 15bf8474b8f93ef22515504ea708fc9e46b91cc3e46a3a97aab5f6038960af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:41:17.264 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:17 smithi156 podman[69529]: 2024-04-25 01:41:17.081774857 +0000 UTC m=+0.018469947 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:41:17.264 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:17 smithi156 podman[69529]: 2024-04-25 01:41:17.185186293 +0000 UTC m=+0.121881416 container create 0031afa976c0025c4623ecebbba4d79033f1477a40c40ba1b77003c4a44d18e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T01:41:17.673 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:17 smithi156 podman[69529]: 2024-04-25 01:41:17.395975661 +0000 UTC m=+0.332670738 container init 0031afa976c0025c4623ecebbba4d79033f1477a40c40ba1b77003c4a44d18e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-25T01:41:17.673 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:17 smithi156 podman[69529]: 2024-04-25 01:41:17.399241109 +0000 UTC m=+0.335936189 container start 0031afa976c0025c4623ecebbba4d79033f1477a40c40ba1b77003c4a44d18e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T01:41:17.673 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:17 smithi156 podman[69529]: 2024-04-25 01:41:17.470929993 +0000 UTC m=+0.407625090 container attach 0031afa976c0025c4623ecebbba4d79033f1477a40c40ba1b77003c4a44d18e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:17.976 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 01:41:17 smithi156 podman[69529]: 2024-04-25 01:41:17.687611495 +0000 UTC m=+0.624306583 container died 0031afa976c0025c4623ecebbba4d79033f1477a40c40ba1b77003c4a44d18e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T01:41:18.192 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T01:41:18.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:18.230 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T01:41:18.230 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T01:41:18.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.a 2024-04-25T01:41:18.617 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:18 smithi016 systemd[1]: Stopping Ceph mds.a for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:19.117 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a[70590]: 2024-04-25T01:41:18.702+0000 7efd08c25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:19.117 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:18 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a[70590]: 2024-04-25T01:41:18.702+0000 7efd08c25700 -1 mds.a *** got signal Terminated *** 2024-04-25T01:41:20.117 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:19 smithi016 podman[95281]: 2024-04-25 01:41:19.851913069 +0000 UTC m=+1.405641595 container died 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:41:20.613 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:20 smithi016 podman[95281]: 2024-04-25 01:41:20.210681305 +0000 UTC m=+1.764409828 container cleanup 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:20.613 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:20 smithi016 bash[95281]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a 2024-04-25T01:41:20.614 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 25 01:41:20 smithi016 podman[95296]: 2024-04-25 01:41:20.420139295 +0000 UTC m=+0.566201347 container remove 318e2f554f042192be3d0a1591542f919373677e186f89081ab28971e884ce52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-25T01:41:20.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T01:41:20.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:20.669 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T01:41:20.669 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T01:41:20.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.d 2024-04-25T01:41:21.017 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:20 smithi016 systemd[1]: Stopping Ceph mds.d for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:21.367 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-d[69631]: 2024-04-25T01:41:21.030+0000 7f7f161f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:21.367 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:21 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-d[69631]: 2024-04-25T01:41:21.030+0000 7f7f161f6700 -1 mds.d *** got signal Terminated *** 2024-04-25T01:41:23.869 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:23 smithi016 podman[95402]: 2024-04-25 01:41:23.576751334 +0000 UTC m=+2.669461941 container died 30acd667c779fca3878b657c567f2c1aab4d4fe2216313cb05391629f5836daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:41:24.234 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:23 smithi016 podman[95402]: 2024-04-25 01:41:23.997790861 +0000 UTC m=+3.090501460 container cleanup 30acd667c779fca3878b657c567f2c1aab4d4fe2216313cb05391629f5836daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:24.234 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:23 smithi016 bash[95402]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-d 2024-04-25T01:41:24.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T01:41:24.498 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:24 smithi016 podman[95416]: 2024-04-25 01:41:24.248862691 +0000 UTC m=+0.671848484 container remove 30acd667c779fca3878b657c567f2c1aab4d4fe2216313cb05391629f5836daf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:24.499 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:24 smithi016 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.d.service: Deactivated successfully. 2024-04-25T01:41:24.499 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:24 smithi016 systemd[1]: Stopped Ceph mds.d for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:41:24.499 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 25 01:41:24 smithi016 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.d.service: Consumed 1.068s CPU time. 2024-04-25T01:41:24.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:24.511 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T01:41:24.511 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T01:41:24.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.g 2024-04-25T01:41:24.862 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:41:24 smithi016 systemd[1]: Stopping Ceph mds.g for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:25.117 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:41:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g[70239]: 2024-04-25T01:41:24.876+0000 7fcc3ea2c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:25.117 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:41:24 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g[70239]: 2024-04-25T01:41:24.876+0000 7fcc3ea2c700 -1 mds.g *** got signal Terminated *** 2024-04-25T01:41:29.138 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:41:28 smithi016 podman[95524]: 2024-04-25 01:41:28.888326993 +0000 UTC m=+4.130762308 container died b451c0c1eadf12944dbe5c9bfb911d4f0209e2c0dffed3fc21d91c4044038e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T01:41:29.505 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:41:29 smithi016 podman[95524]: 2024-04-25 01:41:29.251442396 +0000 UTC m=+4.493877712 container cleanup b451c0c1eadf12944dbe5c9bfb911d4f0209e2c0dffed3fc21d91c4044038e13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T01:41:29.505 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 25 01:41:29 smithi016 bash[95524]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-g 2024-04-25T01:41:29.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T01:41:29.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:29.712 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T01:41:29.712 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T01:41:29.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.j 2024-04-25T01:41:30.024 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:41:29 smithi016 systemd[1]: Stopping Ceph mds.j for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:30.367 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:41:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j[69281]: 2024-04-25T01:41:30.037+0000 7ff69fe06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:30.368 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:41:30 smithi016 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j[69281]: 2024-04-25T01:41:30.038+0000 7ff69fe06700 -1 mds.j *** got signal Terminated *** 2024-04-25T01:41:32.842 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:41:32 smithi016 podman[95646]: 2024-04-25 01:41:32.564180886 +0000 UTC m=+2.617443151 container died 8d26326a4cc66fc870eef59cbb7c2f34ea1b1713c06885d3ee04ff2fab2006b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:41:33.117 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:41:32 smithi016 podman[95646]: 2024-04-25 01:41:32.946993759 +0000 UTC m=+3.000256025 container cleanup 8d26326a4cc66fc870eef59cbb7c2f34ea1b1713c06885d3ee04ff2fab2006b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:41:33.117 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 25 01:41:32 smithi016 bash[95646]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-j 2024-04-25T01:41:33.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T01:41:33.399 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:33.399 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T01:41:33.399 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T01:41:33.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.b 2024-04-25T01:41:33.723 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:33 smithi038 systemd[1]: Stopping Ceph mds.b for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:34.223 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: 2024-04-25T01:41:33.737+0000 7fb1c41b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:34.223 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:33 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b[62867]: 2024-04-25T01:41:33.737+0000 7fb1c41b3700 -1 mds.b *** got signal Terminated *** 2024-04-25T01:41:37.842 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:37 smithi038 podman[68478]: 2024-04-25 01:41:37.568199064 +0000 UTC m=+3.943574378 container died 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T01:41:38.177 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:37 smithi038 podman[68478]: 2024-04-25 01:41:37.97925098 +0000 UTC m=+4.354626305 container cleanup 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T01:41:38.177 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:37 smithi038 bash[68478]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b 2024-04-25T01:41:38.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T01:41:38.427 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:38 smithi038 podman[68505]: 2024-04-25 01:41:38.189507026 +0000 UTC m=+0.622573430 container remove 305e466c097baadfa859bf7234240ac76138b15b0364413b8e6f6562609fb956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:41:38.427 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:38 smithi038 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-25T01:41:38.428 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:38 smithi038 systemd[1]: Stopped Ceph mds.b for 605c42f8-02a1-11ef-bc93-c7b262605968. 2024-04-25T01:41:38.428 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 25 01:41:38 smithi038 systemd[1]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.b.service: Consumed 6.528s CPU time. 2024-04-25T01:41:38.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:38.441 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T01:41:38.441 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T01:41:38.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.e 2024-04-25T01:41:38.723 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:41:38 smithi038 systemd[1]: Stopping Ceph mds.e for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:39.223 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:41:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e[62242]: 2024-04-25T01:41:38.808+0000 7f0d85fe8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:39.224 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:41:38 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e[62242]: 2024-04-25T01:41:38.808+0000 7f0d85fe8700 -1 mds.e *** got signal Terminated *** 2024-04-25T01:41:41.814 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:41:41 smithi038 podman[68611]: 2024-04-25 01:41:41.560454907 +0000 UTC m=+2.883927226 container died 6ffe41c2eb68c3f63b990b221c57b59916bbd003808cb8b009bdc666ebaa0a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9) 2024-04-25T01:41:42.172 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:41:41 smithi038 podman[68611]: 2024-04-25 01:41:41.926146427 +0000 UTC m=+3.249618745 container cleanup 6ffe41c2eb68c3f63b990b221c57b59916bbd003808cb8b009bdc666ebaa0a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:41:42.172 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 25 01:41:41 smithi038 bash[68611]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-e 2024-04-25T01:41:42.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T01:41:42.330 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:42.330 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T01:41:42.330 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T01:41:42.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.h 2024-04-25T01:41:42.473 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:41:42 smithi038 systemd[1]: Stopping Ceph mds.h for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:42.973 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:41:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h[62555]: 2024-04-25T01:41:42.671+0000 7f8cc98a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:42.974 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:41:42 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h[62555]: 2024-04-25T01:41:42.671+0000 7f8cc98a5700 -1 mds.h *** got signal Terminated *** 2024-04-25T01:41:45.723 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:41:45 smithi038 podman[68732]: 2024-04-25 01:41:45.565380883 +0000 UTC m=+3.003185276 container died 0bc659dae5f40df83c01be0310090c79a85aed9f3769d78cef48a57dcb7ace0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T01:41:46.223 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:41:46 smithi038 podman[68732]: 2024-04-25 01:41:46.047200297 +0000 UTC m=+3.485004690 container cleanup 0bc659dae5f40df83c01be0310090c79a85aed9f3769d78cef48a57dcb7ace0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T01:41:46.224 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 25 01:41:46 smithi038 bash[68732]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-h 2024-04-25T01:41:46.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T01:41:46.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:46.445 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T01:41:46.445 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T01:41:46.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.k 2024-04-25T01:41:46.723 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:41:46 smithi038 systemd[1]: Stopping Ceph mds.k for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:47.223 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:41:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k[61924]: 2024-04-25T01:41:46.767+0000 7f7f1d01b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:47.224 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:41:46 smithi038 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k[61924]: 2024-04-25T01:41:46.767+0000 7f7f1d01b700 -1 mds.k *** got signal Terminated *** 2024-04-25T01:41:49.955 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:41:49 smithi038 podman[68851]: 2024-04-25 01:41:49.594406829 +0000 UTC m=+2.914740053 container died b423c9ba0b88e50b7b10fcbec36bc9beb5e5ea7f8eb3795b5e3e855baf846b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-25T01:41:50.223 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:41:50 smithi038 podman[68851]: 2024-04-25 01:41:50.064673486 +0000 UTC m=+3.385006712 container cleanup b423c9ba0b88e50b7b10fcbec36bc9beb5e5ea7f8eb3795b5e3e855baf846b5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:41:50.224 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 25 01:41:50 smithi038 bash[68851]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-k 2024-04-25T01:41:50.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T01:41:50.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:50.557 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T01:41:50.557 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T01:41:50.558 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.c 2024-04-25T01:41:50.845 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:41:50 smithi156 systemd[1]: Stopping Ceph mds.c for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:51.258 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:41:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c[63622]: 2024-04-25T01:41:50.857+0000 7fa4673b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:51.258 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:41:50 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c[63622]: 2024-04-25T01:41:50.857+0000 7fa4673b7700 -1 mds.c *** got signal Terminated *** 2024-04-25T01:41:53.820 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:41:53 smithi156 podman[69645]: 2024-04-25 01:41:53.562987012 +0000 UTC m=+2.791308943 container died 9efd5d18a612b8340f6e40068343d4787e5bc67f45fe8f082b34b75ac330e360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T01:41:54.154 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:41:53 smithi156 podman[69645]: 2024-04-25 01:41:53.932427945 +0000 UTC m=+3.160749867 container cleanup 9efd5d18a612b8340f6e40068343d4787e5bc67f45fe8f082b34b75ac330e360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True) 2024-04-25T01:41:54.154 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 01:41:53 smithi156 bash[69645]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-c 2024-04-25T01:41:54.306 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T01:41:54.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:54.341 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T01:41:54.341 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T01:41:54.341 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.f 2024-04-25T01:41:54.758 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:41:54 smithi156 systemd[1]: Stopping Ceph mds.f for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:54.758 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:41:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f[63314]: 2024-04-25T01:41:54.668+0000 7f5dcd39b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:54.759 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:41:54 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f[63314]: 2024-04-25T01:41:54.668+0000 7f5dcd39b700 -1 mds.f *** got signal Terminated *** 2024-04-25T01:41:57.823 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:41:57 smithi156 podman[69778]: 2024-04-25 01:41:57.568913707 +0000 UTC m=+3.001623598 container died 554da208df0b26296c0efce77ffc6bc4a4f41871389ddda54dfe869d82b72b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T01:41:58.199 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:41:57 smithi156 podman[69778]: 2024-04-25 01:41:57.927840582 +0000 UTC m=+3.360550424 container cleanup 554da208df0b26296c0efce77ffc6bc4a4f41871389ddda54dfe869d82b72b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T01:41:58.199 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 01:41:57 smithi156 bash[69778]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-f 2024-04-25T01:41:58.333 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T01:41:58.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:41:58.365 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T01:41:58.365 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T01:41:58.366 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.i 2024-04-25T01:41:58.759 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:41:58 smithi156 systemd[1]: Stopping Ceph mds.i for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:41:58.759 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:41:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i[63930]: 2024-04-25T01:41:58.705+0000 7fe9e7d4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:41:58.759 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:41:58 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i[63930]: 2024-04-25T01:41:58.705+0000 7fe9e7d4f700 -1 mds.i *** got signal Terminated *** 2024-04-25T01:42:01.852 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:42:01 smithi156 podman[69897]: 2024-04-25 01:42:01.573224731 +0000 UTC m=+2.974496272 container died 492b7fcb72ca84f94dd0a8c40baf14acf3cce8bdf8e56739332e41fd565705db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T01:42:02.170 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:42:01 smithi156 podman[69897]: 2024-04-25 01:42:01.95658748 +0000 UTC m=+3.357859030 container cleanup 492b7fcb72ca84f94dd0a8c40baf14acf3cce8bdf8e56739332e41fd565705db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T01:42:02.171 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 01:42:01 smithi156 bash[69897]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-i 2024-04-25T01:42:02.291 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T01:42:02.322 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:42:02.322 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T01:42:02.322 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T01:42:02.322 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.l 2024-04-25T01:42:02.508 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:42:02 smithi156 systemd[1]: Stopping Ceph mds.l for 605c42f8-02a1-11ef-bc93-c7b262605968... 2024-04-25T01:42:03.008 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:42:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l[63012]: 2024-04-25T01:42:02.751+0000 7f1d3bc14700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:42:03.009 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:42:02 smithi156 ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l[63012]: 2024-04-25T01:42:02.751+0000 7f1d3bc14700 -1 mds.l *** got signal Terminated *** 2024-04-25T01:42:06.848 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:42:06 smithi156 podman[70017]: 2024-04-25 01:42:06.580715608 +0000 UTC m=+4.031710742 container died 4cf67c1f405b13ebd9e155b5ac6ba4d9a890424d08b9f9c1f7f67ad480343bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T01:42:06.848 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:42:06 smithi156 podman[70017]: 2024-04-25 01:42:06.861063563 +0000 UTC m=+4.312058692 container cleanup 4cf67c1f405b13ebd9e155b5ac6ba4d9a890424d08b9f9c1f7f67ad480343bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:42:07.099 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 01:42:06 smithi156 bash[70017]: ceph-605c42f8-02a1-11ef-bc93-c7b262605968-mds-l 2024-04-25T01:42:07.225 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-605c42f8-02a1-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T01:42:07.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:42:07.263 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T01:42:07.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:42:07.471 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:42:10.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:42:10.403 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:42:12.964 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:42:13.152 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:42:15.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:42:15.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:42:15.585 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:42:15.610 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T01:42:15.612 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi016/crash 2024-04-25T01:42:15.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:42:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:42:15.649 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi038/crash 2024-04-25T01:42:15.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:42:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:42:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:42:15.692 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi156/crash 2024-04-25T01:42:15.693 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:42:15.743 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:42:15.743 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:42:15.744 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T01:42:15.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-25T01:42:15.854 INFO:tasks.cephadm:Compressing logs... 2024-04-25T01:42:15.855 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:42:15.857 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:42:15.860 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:42:15.883 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-25T01:42:15.883 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-04-25T01:42:15.883 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T01:42:15.884 INFO:teuthology.orchestra.run.smithi156.stderr:gzip/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T01:42:15.887 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr: 86.8% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.3% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:42:15.888 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T01:42:15.890 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:42:15.890 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T01:42:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T01:42:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T01:42:15.893 INFO:teuthology.orchestra.run.smithi038.stderr: 86.8% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:42:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.3% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:42:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:42:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T01:42:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:42:15.897 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.11.log: 93.0% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T01:42:15.898 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:42:15.901 INFO:teuthology.orchestra.run.smithi038.stderr: 92.9% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T01:42:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T01:42:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.g.log: /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr: 86.8% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:42:15.902 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T01:42:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T01:42:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-25T01:42:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:42:15.904 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T01:42:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:42:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T01:42:15.912 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:42:15.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T01:42:15.920 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:42:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:42:15.929 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 94.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T01:42:15.935 INFO:teuthology.orchestra.run.smithi156.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:42:15.935 INFO:teuthology.orchestra.run.smithi156.stderr: 91.0% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:42:15.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 94.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T01:42:15.946 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:42:15.950 INFO:teuthology.orchestra.run.smithi156.stderr: 93.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T01:42:15.955 INFO:teuthology.orchestra.run.smithi038.stderr: 93.6% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T01:42:15.957 INFO:teuthology.orchestra.run.smithi156.stderr: 94.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T01:42:15.977 INFO:teuthology.orchestra.run.smithi038.stderr: 94.6% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T01:42:15.988 INFO:teuthology.orchestra.run.smithi156.stderr: 93.5% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T01:42:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: 90.1% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:42:16.088 INFO:teuthology.orchestra.run.smithi016.stderr: 94.6% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T01:42:16.144 INFO:teuthology.orchestra.run.smithi016.stderr: 94.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T01:42:16.146 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:42:16.165 INFO:teuthology.orchestra.run.smithi156.stderr: 91.9% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T01:42:16.196 INFO:teuthology.orchestra.run.smithi016.stderr: 94.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T01:42:16.206 INFO:teuthology.orchestra.run.smithi016.stderr: 94.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T01:42:16.229 INFO:teuthology.orchestra.run.smithi016.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:42:16.260 INFO:teuthology.orchestra.run.smithi016.stderr: 82.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:42:16.318 INFO:teuthology.orchestra.run.smithi038.stderr: 91.9% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T01:42:16.384 INFO:teuthology.orchestra.run.smithi038.stderr: 92.8% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T01:42:17.784 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T01:42:29.944 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T01:42:52.023 INFO:teuthology.orchestra.run.smithi038.stderr: 91.6% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T01:42:54.010 INFO:teuthology.orchestra.run.smithi156.stderr: 90.8% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T01:42:55.591 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T01:42:56.444 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T01:43:00.185 INFO:teuthology.orchestra.run.smithi038.stderr: 90.9% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T01:43:02.632 INFO:teuthology.orchestra.run.smithi038.stderr: 90.6% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T01:43:04.849 INFO:teuthology.orchestra.run.smithi038.stderr: 90.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T01:43:04.850 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-25T01:43:04.851 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m48.974s 2024-04-25T01:43:04.851 INFO:teuthology.orchestra.run.smithi038.stderr:user 2m51.784s 2024-04-25T01:43:04.851 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m4.931s 2024-04-25T01:43:04.982 INFO:teuthology.orchestra.run.smithi156.stderr: 90.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T01:43:04.983 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-04-25T01:43:04.984 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m49.108s 2024-04-25T01:43:04.984 INFO:teuthology.orchestra.run.smithi156.stderr:user 2m42.481s 2024-04-25T01:43:04.984 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m5.205s 2024-04-25T01:47:04.582 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T01:48:37.983 INFO:teuthology.orchestra.run.smithi016.stderr: 90.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T01:50:05.830 INFO:teuthology.orchestra.run.smithi016.stderr: 90.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T01:50:47.157 INFO:teuthology.orchestra.run.smithi016.stderr: 90.7% -- replaced with /var/log/ceph/605c42f8-02a1-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T01:52:47.090 INFO:teuthology.orchestra.run.smithi016.stderr: 94.3% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-25T01:52:47.091 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-25T01:52:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:real 10m31.208s 2024-04-25T01:52:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:user 5m30.390s 2024-04-25T01:52:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m13.297s 2024-04-25T01:52:47.092 INFO:tasks.cephadm:Archiving logs... 2024-04-25T01:52:47.092 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi016/log 2024-04-25T01:52:47.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:53:37.932 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi038/log 2024-04-25T01:53:37.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:54:08.516 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi156/log 2024-04-25T01:54:08.517 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:54:38.345 INFO:tasks.cephadm:Removing cluster... 2024-04-25T01:54:38.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --force 2024-04-25T01:54:38.558 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:54:39.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --force 2024-04-25T01:54:39.246 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:54:39.812 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 605c42f8-02a1-11ef-bc93-c7b262605968 --force 2024-04-25T01:54:40.004 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 605c42f8-02a1-11ef-bc93-c7b262605968 2024-04-25T01:54:40.558 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T01:54:40.559 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:54:40.578 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:54:40.598 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:54:40.613 INFO:tasks.cephadm:Teardown complete 2024-04-25T01:54:40.614 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T01:54:40.626 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T01:54:40.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:54:40.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:54:40.640 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:54:40.748 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T01:54:40.758 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T01:54:40.758 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:54:40.761 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:54:40.763 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:54:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:54:40.777 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:54:40.778 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:54:40.780 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:54:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-25T01:54:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:^- hv01.front.sepia.ceph.com 2 6 377 51 +1597us[+1597us] +/- 570ms 2024-04-25T01:54:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -272us[ -273us] +/- 74ms 2024-04-25T01:54:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +292us[ +291us] +/- 81ms 2024-04-25T01:54:40.781 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:54:40.782 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:54:40.782 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-25T01:54:40.783 INFO:teuthology.orchestra.run.smithi156.stdout:^- hv01.front.sepia.ceph.com 2 6 377 48 +1598us[+1598us] +/- 571ms 2024-04-25T01:54:40.783 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -261us[ -248us] +/- 74ms 2024-04-25T01:54:40.783 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +297us[ +297us] +/- 82ms 2024-04-25T01:54:40.783 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:54:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:54:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-25T01:54:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:^- hv01.front.sepia.ceph.com 2 6 377 48 +1653us[+1653us] +/- 571ms 2024-04-25T01:54:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -265us[ -258us] +/- 75ms 2024-04-25T01:54:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +306us[ +306us] +/- 82ms 2024-04-25T01:54:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:54:40.784 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T01:54:40.797 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T01:54:40.798 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T01:54:40.808 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:54:40.933 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:54:41.006 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:54:41.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:54:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:41.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:54:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:41.131 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:54:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:41.157 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T01:54:41.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T01:54:41.179 INFO:teuthology.task.internal:Duration was 2668.264414 seconds 2024-04-25T01:54:41.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T01:54:41.190 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T01:54:41.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:54:41.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:54:41.196 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:54:41.233 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:54:41.234 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:54:41.258 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:54:41.567 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T01:54:41.567 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-25T01:54:41.567 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:54:41.591 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-25T01:54:41.591 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:54:41.617 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-04-25T01:54:41.618 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:54:41.640 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T01:54:41.640 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:54:41.643 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:54:41.660 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:54:41.705 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T01:54:41.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:54:41.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:54:41.728 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:54:42.098 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T01:54:42.115 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T01:54:42.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:54:42.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:54:42.206 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:54:42.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T01:54:42.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:54:42.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:54:42.265 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:54:42.286 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-25T01:54:42.292 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-25T01:54:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-04-25T01:54:42.322 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:54:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:42.365 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:54:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:42.380 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:54:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:42.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T01:54:42.408 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T01:54:42.411 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi016 2024-04-25T01:54:42.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:54:42.460 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi038 2024-04-25T01:54:42.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:54:42.497 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672312/remote/smithi156 2024-04-25T01:54:42.498 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:54:42.533 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T01:54:42.533 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:54:42.535 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:54:42.537 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:54:42.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T01:54:42.595 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T01:54:42.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T01:54:42.606 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T01:54:42.606 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:54:42.609 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:54:42.611 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:54:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:54 /home/ubuntu/cephtest 2024-04-25T01:54:42.624 INFO:teuthology.orchestra.run.smithi038.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:54 /home/ubuntu/cephtest 2024-04-25T01:54:42.636 INFO:teuthology.orchestra.run.smithi156.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:54 /home/ubuntu/cephtest 2024-04-25T01:54:42.638 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T01:54:42.649 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T01:54:42.676 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} duration: 2668.2644143104553 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T01:54:42.676 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:54:42.752 INFO:teuthology.run:pass